different crates can be compiled in parallel

different crates make it easier to build on top of for other tools, for one, without having to depend on all of the pants rust code. it would have been much more difficult to build the thing i’m working on now if it was a single crate

@hundreds-father-404: so… i’m deep on this cpython crate stuff. am pretty close to the finish line, so not going to give up today. but beyond the linker error i’m looking at (working on reproing locally in docker), i just also noticed that the crate doesn’t use PEP-384 (although it has nascent support for it)

For Pants to be truly extensible even more extensible, I think we’ll need a mechanism to deregister rules? Lets say you don’t like how Pants’ builtin rule works for prepare_python_sources. (Maybe you want to disable injecting missing

__init__.py

files, for example). You want to use all of the core Python implementation, but want to swap out the rule that goes from

ImportablePythonSourcesRequest -> ImportanlePythonSources

with your own with the same signature. You can’t just make your own due to ambiguity, you need to first deregister the core rule. We allow this type of extensibility with the Target API, but not yet the Rules API. Should be very trivial to implement. Main questions are a) do we indeed want this, and b) how do we expose it?

“truly extensible” feels like mildly aggressive wording. could possibly be phrased as “i was thinking recently about pants and extensibility [or provide some other context] and i realized v2 rules don’t have a flexibility that v1 rules do” or something

i think deregistering rules makes perfect sense and i think the v2 api makes this a LOT cleaner than the v1 version

you could explicitly specify which rules you want to deregister (as one example) and then you know exactly which parts of the graph you need to reconstruct (possibly we could make tooling to help visualize this?)

another possible interface is “i know i want this new rule i just made to ALWAYS run to convert A -> B, and just disregard whatever else was previously registered for A if the product is intended to be B”

so if the engine previously had (A -> D -> E -> B), where A->D, D->E, and E->B are distinct `@rule`s

we could add some logic possibly in

rules.py

OR in the rust code (@witty-crayon-22786 has convinced me that it's often a great idea to keep meta-logic like this in python whenever possible) to then say "if the subject is an A, and B is within the required path, use my new rule instead"

these could both be implemented separately

i think we could possibly make an interface for version 2 that would be really nice and take advantage of the engine's type safety to avoid causing issues if the feature is used often

Interesting. With the second proposal, I agree it would be better to implant that in python than rust to avoid making the rust code for complex. The deregistering approach seems simpler, but worth considering both.

it feels safer to me for eventually stabilizing the engine if we don't have people working with code that's literally removing rules from the rule graph

that's a totally subjective judgement

but also, i agree that version 1 seems simpler to reason about atm, which seems like a good thing. could also be implemented in

rules.py

or around there

hm. we could also try something like "swap exactly this one rule with _exactly this one other new rule i made_"

one thing i like that we've done is to keep our

@rule

methods as public (without underscores), unlike bazel, which does the exact opposite

it makes things like "operate on this exact rule defined in base pants" more idiomatic (just import the rule)

(1) allow de-registering specific `@rule`s (not intrinsics) (2) allow "patching" specific A -> B subgraphs spanned by a specific new rule or set of rules (3) swap out exactly one rule with exactly another, with the same params and return type

i think this is a really good line of investigation btw and i'm super glad you brought it up

and we think these all could be done in python too. not that it would be harder in rust, but that's just where i think we could look first to implement this

also, wrt this: https://pantsbuild.slack.com/archives/C0D7TNJHL/p1587950091344000 i think (2) could be a reasonable eventual followup to (3), and (3) seems the most stable/easiest to reason about for me atm. just imho

I've been trying ot make the workunit abstraction do teh right thing in terms of recording node start times for a few days now

I have a thing compiling, but I had to bump https://doc.rust-lang.org/reference/attributes/limits.html#the-type_length_limit-attribute

to something like 7 million. I"m not sure what this number means or how big we can get away with setting it

people in this hn thread are gushing about rust-analyzer: https://news.ycombinator.com/item?id=22995466

what the heck… how long has this existed ? https://github.com/pantsbuild/pants/blob/fdc097576a0ef0dd1d3eeb0ebb535bf1b2da3887/3rdparty/protobuf/bazelbuild_remote-apis/build/bazel/remote/execution/v2/remote_execution.proto#L346-L362

See https://pantsbuild.slack.com/archives/C012CFZAK0B/p1588018971011100 for discussion on making subsystems extensible to plugin authors, i.e. you can add new options to pre-existing subsystems