I'm looking to port "core" field types to rust. Th...
# developmentw
worried-painter-31382
03/31/2026, 8:59 AMI'm looking to port "core" field types to rust. The end goal is moving source root and target hydration to rust. It is a 20% runtime win across all goals in my work repo (using a dirty hack that just short circuits to native rust for the numbers).
I have confirmed with ugly hacks it brings me ~20% improved pants runtime in my work repo. More modest in pants repo but it will be net faster. I dont think it is a controversal migration (it's piggybacking on stu's work), but this is a headsup if we have reservations.
🔥 1
worried-painter-31382
03/31/2026, 9:01 AMNet effect is if you want to modify core logic in field bases like BoolField or StringSequenceField you have to code in rust.
w
wide-midnight-78598
03/31/2026, 10:32 AMIs there any API change/breakage for in-repo plugins?
That's usually my only question when it comes to large changes
w
worried-painter-31382
03/31/2026, 10:42 AMNah, it'll be samesies. Subclassing is unaffected
🎉 1
c
curved-manchester-66006
03/31/2026, 3:23 PMSo, this is extremely rough vibe code, but I guess I have to
pushperfdependencies ::frozendict Copy code
$ ~/src/o/pants-scratch-perf-scripts/py_class_profile.py --module='*' deps-script.profile.txt
Process filter: tokio-runtime-w
Module filter: *
Stacks: 87652 total, 23415 with py::, 23415 with matching module
Self % Total % Class (file)
------ ------ ------ ------ ---
5139 21.9% 6343 27.1% _rule_call_trampoline (pants/engine/rules.py)
5139 / 6343 <locals>.wrapper
2437 10.4% 7237 30.9% (free) (pants/engine/internals/graph.py)
643 / 2533 resolve_dependencies
553 / 1087 determine_explicitly_provided_dependencies
437 / 948 hydrate_sources
168 / 533 resolve_targets
163 / 178 convert_dependencies_request_to_explicitly_provided_dependencies_request
1377 5.9% 3242 13.8% (free) (pants/backend/python/dependency_inference/rules.py)
356 / 885 infer_python_init_dependencies
335 / 1353 infer_python_dependencies_via_source
269 / 1004 resolve_parsed_dependencies
251 / 492 _exec_parse_deps
116 / 155 _collect_imports_info
1374 5.9% 1400 6.0% _Concurrently (pants/engine/internals/selectors.py)
1374 / 1400 __await__
1361 5.8% 1996 8.5% (free) (<string>)
578 / 859 __create_fn__.<locals>.__init__
562 / 892 __create_fn__.<locals>.__hash__
221 / 245 __create_fn__.<locals>.__eq__
804 3.4% 816 3.5% PurePath (stdlib:pathlib/__init__.py)
108 / 108 _parse_path
91 / 278 __hash__
88 / 191 _str_normcase
70 / 145 drive
63 / 117 _tail
767 3.3% 1467 6.3% Target (pants/engine/target.py)
144 / 513 _calculate_field_values
119 / 455 _has_fields
100 / 619 has_fields
85 / 97 __hash__
70 / 142 _find_registered_field_subclass.<locals>.<genexpr>
664 2.8% 832 3.6% FrozenDict (pants/util/frozendict.py)
212 / 563 __init__
174 / 251 _calculate_hash
172 / 173 __getitem__
67 / 67 __iter__
22 / 22 __hash__
523 2.2% 715 3.1% _OptionBase (pants/option/option_types.py)
482 / 715 __get__
40 / 51 _convert_
1 / 3 get_flag_options
519 2.2% 1183 5.1% (free) (pants/core/util_rules/stripped_source_files.py)
366 / 750 strip_source_roots
153 / 433 strip_file_name
506 2.2% 3687 15.7% (free) (pants/engine/internals/selectors.py)
506 / 3687 Concurrently
454 1.9% 1071 4.6% (free) (pants/source/source_root.py)
251 / 373 get_source_roots
177 / 613 get_optional_source_roots
13 / 20 get_source_root
13 / 65 get_optional_source_root
433 1.8% 957 4.1% (free) (pants/backend/python/dependency_inference/parse_python_dependencies.py)
433 / 957 parse_python_dependencies
422 1.8% 739 3.2% (free) (pants/backend/python/util_rules/ancestor_files.py)
351 / 739 find_ancestor_files
71 / 156 putative_ancestor_files
377 1.6% 433 1.8% (free) (pants/engine/intrinsics.py)
149 / 205 parse_python_deps
84 / 84 digest_to_snapshot
74 / 74 path_globs_to_digest
44 / 44 merge_digests
18 / 18 get_digest_contents
306 1.3% 613 2.6% (free) (pants/core/util_rules/source_files.py)
306 / 613 determine_source_files
291 1.2% 343 1.5% AsyncFieldMixin (pants/engine/target.py)
236 / 237 __hash__
55 / 106 __new__
247 1.1% 247 1.1% Call (pants/engine/internals/selectors.py)
247 / 247 __await__
247 1.1% 403 1.7% Mapping (<frozen _collections_abc>)
72 / 91 keys
72 / 129 __contains__
56 / 101 get
31 / 56 items
16 / 26 values
230 1.0% 605 2.6% memoized (pants/util/memo.py)
230 / 605 <locals>.memoize
228 1.0% 493 2.1% SourcesField (pants/engine/target.py)
81 / 190 path_globs
61 / 222 validate_resolved_files
30 / 44 filespec_matcher
22 / 40 filespec
15 / 38 prefix_glob_with_dirpath
226 1.0% 1653 7.1% (free) (pants/backend/python/target_types_rules.py)
214 / 1612 validate_python_dependencies
5 / 21 resolve_pex_entry_point
5 / 10 python_files_generator_settings
1 / 1 infer_python_distribution_dependencies
1 / 8 infer_pex_binary_entry_point_dependency
202 0.9% 202 0.9% OptionValueContainer (pants/option/option_value_container.py)
81 / 81 _get_underlying_value
72 / 153 __getattr__
41 / 49 is_flagged
8 / 8 get_rank
200 0.9% 243 1.0% validate_python_dependencies (pants/backend/python/target_types_rules.py)
200 / 243 <locals>.<genexpr>
196 0.8% 196 0.8% ABCMeta (<frozen abc>)
143 / 144 __instancecheck__
53 / 53 __subclasscheck__
185 0.8% 190 0.8% _AbstractOrderedSet (pants/util/ordered_set.py)
79 / 88 __init__
44 / 83 union
28 / 28 __iter__
25 / 25 __contains__
8 / 8 __len__
181 0.8% 226 1.0% (free) (pants/util/memo.py)
157 / 222 per_instance
24 / 24 equal_args
179 0.8% 240 1.0% resolve_parsed_dependencies (pants/backend/python/dependency_inference/rules.py)
179 / 240 <locals>.<genexpr>
167 0.7% 168 0.7% (free) (<frozen posixpath>)
97 / 114 join
32 / 34 dirname
19 / 19 _get_sep
13 / 13 splitdrive
4 / 6 relpath
164 0.7% 209 0.9% resolve_unexpanded_targets (pants/engine/internals/graph.py)
164 / 209 <locals>.<genexpr>
151 0.6% 723 3.1% resolve_dependencies (pants/engine/internals/graph.py)
151 / 723 <locals>.<genexpr>
137 0.6% 307 1.3% KeysView (<frozen _collections_abc>)
89 / 130 __iter__
48 / 177 __contains__
111 0.5% 168 0.7% PathGlobs (pants/engine/fs.py)
103 / 168 __init__
8 / 8 __post_init__
98 0.4% 316 1.3% (free) (pants/backend/python/dependency_inference/module_mapper.py)
85 / 177 map_module_to_address
7 / 81 map_first_party_python_targets_to_modules
3 / 50 find_all_python_projects
2 / 12 module_from_stripped_path
1 / 7 merge_first_party_module_mappings
90 0.4% 648 2.8% (free) (pants/engine/internals/build_files.py)
33 / 59 find_build_file
30 / 556 parse_address_family
21 / 27 find_target_adaptor
6 / 6 ensure_address_family
0 / 2 _get_target_adaptorw
worried-painter-31382
03/31/2026, 3:35 PMAye it helps validating what I already am experimenting with. Which is to expand the engine to allow native rules ("intrinsics") to never leave rust when they invoke eachother:
https://github.com/tobni/pants/tree/add/inline-native-calls
That would dodge the trampoline. Dodging the trampoline does require chains of rules to have purely "native" (as in rust only) layout.
To do that the data model has to be rust-first though. As soon as we have to go back to python land we'll get overhead per task. GIL make this serial, but that might be fixed using free-threaded python. Rust binary will always be faster than python bytecode though, so I think the approach of "fusing" native calls has merit regardless.
worried-painter-31382
03/31/2026, 3:41 PMpants is slow for two reasons:
1. Hard work is done in python
2. Every task needs to ping back to python to proceed
1 you fix by migrating work to rust. 2 you fix by making the engine smarter OR by writing rules smarter. I am more interested in 1 and 2a, as 2b will have plugin implications.
worried-painter-31382
03/31/2026, 3:46 PMAs an example for 2b vs 2a: if you send ALL python files that needs dep inference parsing as one task instead of sending one task per file, you'll decimate the runtime of
pants dependencies ::h
happy-kitchen-89482
03/31/2026, 5:51 PMI am in favor of porting anything to Rust if it improves perf, so port away!
happy-kitchen-89482
03/31/2026, 5:53 PMAlso, note that with https://github.com/pantsbuild/pants/pull/23075 and https://github.com/pantsbuild/pants/pull/23076 we can send all dep inference requests in a single task, and it doesn't affect caching because native dep inference caching is special-cased and is not done via Process caching
happy-kitchen-89482
03/31/2026, 5:53 PMI am using this in the ng python backend, haven't yet gotten around to integrating it in the existing backends, but it shouldn't be too crazy
w
worried-painter-31382
03/31/2026, 5:55 PMUnfortunate that I hit a special case in my example! I think the point still stands if we want to truly be fast, the engine needs work, not juts plugins. I'm ok to be wrong though
h
happy-kitchen-89482
03/31/2026, 5:56 PMOh for sure, just saying that I approve of both porting to rust and hacking on the engine to improve perf, and that this has already been done in the one case you mentioned, but there are many others
happy-kitchen-89482
03/31/2026, 5:56 PMIn case that was a case you particularly care about
👍 1
w
worried-painter-31382
03/31/2026, 8:27 PMOn that note: https://github.com/pantsbuild/pants/pull/23204
worried-painter-31382
04/16/2026, 10:40 PMSome engine hacking: https://github.com/pantsbuild/pants/pull/23268
🔥 2
2 Views