I just wanted to comment here to say that we've given up on Pants for our Scala monorepo for now. It's missing basic functionality, like publishing the compiled code as a jar to a Maven repository. The deploy_jar target doesn't work and creates jars that won't run due to some jar signature issues. That all wouldn't be so bad if it were easy to hack on so we could add what we need ourselves, but it's not. The whole rule graph thing makes it extremely hard to follow the control flow because as soon as you hit an

await Get

line, you don't know which rule is being invoked next. I have no idea how I'm supposed to figure out how all the different rules fit together. Yes, you can generate rule graph PDFs. Those things are the closest thing to actual spaghetti that I've ever seen in programming, and as far as I can see they don't even include the names of the rules that were invoked.

Sounds like you made a serious effort to make it work here. Kudos. I agree there’s a learning curve to get into how things work under the hood in pants (and there’s always room for improvements). What I tend to do however is not to understand the whole graph (as you say, that gets very complicatged/involved and is hard to reason about) but I find it shines when you look at just isolated pieces, like for a single

Get

it’s fairly straight forward to figure out which rule is being invoked once you’ve gotten into it as it’s all type based on inputs and outputs. Best of luck and welcome back any time.

Well obviously you can't understand the whole thing at once, that's true of any kind of programming.

Too bad Pants wasn't a better fit for your needs. The Scala backend could use some love, it's true. FWIW, this proposal would move us away from

Get

-by-type to call-by-name conventions. We do plan to implement this soon, and it would make following the rule graph much more straightforward (as well as have performance benefits).

That sounds like a great improvement

I've been studying the code and I found similar issues, reminiscent of the complications with understanding how an application gets built when using dependency injection frameworks, combined with the complications in debugging through async/callback code. But in digging around and poking at it, it looks like there could be opportunities to make rule execution easier to understand during development even within the current

Get

framework - tooling in addition to the rule graph pdf, which I too tried and found unwieldy (as has often been the case when visualizing a big graph in 2d without a way to interactively manage the level of detail: it becomes spaghetti). I played with things like having the program print the rule it was running at any given time (by logging from within

selectors.native_engine_generator_send

after it hits a certain rule) which seems like could allow for a linear interpretation of the execution flow with scale management if we can keep track of which rules got called because of Gets issued within a caller rule. I was hunting for something very simple - like to find out where in the code we are intentionally dropping

file

targets when building pex binaries, but it was a struggle to attempt to debug through.

fwiw, this got me curious to find out how difficult it would be to find out given a good understanding of how pants rules works. This is a not well-known area of the code base for me, so I know very little about the particulars and as such gives a pretty good metric for how bad the nest of spaghetti rules situation is (or not). It took me roughly 10 minutes to find out how and where

file

targets are filtered out for

pex_binary

targets. I’ll drop links in reverse order for the crumbs I followed to arrive there (starting with the final location): • https://github.com/pantsbuild/pants/blob/4f13ae2f02823fb198c0e0c10538f6423ff4dc04/src/python/pants/backend/python/util_rules/python_sources.py#L[…]3 This collects all

resource

targets (or technically, all targets with a

ResourceSourceField

) • https://github.com/pantsbuild/pants/blob/4f13ae2f02823fb198c0e0c10538f6423ff4dc04/src/python/pants/backend/python/util_rules/pex_from_targets.py#L551 The defaults for the

PythonSourceFilesRequest

is to include

resource

and exclude

file

targets. • https://github.com/pantsbuild/pants/blob/4f13ae2f02823fb198c0e0c10538f6423ff4dc04/src/python/pants/backend/python/goals/package_pex_binary.py#L123 I jumped from here to the

create_pex_from_targets

rule. I started here as the entry point for packaging pex binaries. (found it by expecting it would be something like it around here somewhere..) perhaps interesting to some..?

@curved-television-6568, It's easier to find what you're looking for when you are generally experienced with the code base, the language and the tooling around it. And I'm going to admit that I struggle dealing with Python code because I just don't like Python

absolutely, that was part of my point, to get a sense of how big difference it would be going into it with more knowledge about the general structure..

I see

I fully agree with you that the current state of python type checking is lacking in many regards (of course there’s reasons, but doesn’t change what we have)

Sure