I do feel like I see more "things" output on the dyanmic UI than I see in the workunits callback

It's also possible my callback code is wrong 🙂

No dice.

Here's what I see when linting a single file (FWIW I use

flake8

,

pylint

,

isort

, and

yapf

. But I only see Pylint)

It's also possible my callback code is wrong

Thats it 🙂

Much more colorful 🙂

Screenshot from 2021-12-22 09-40-44.png

However, I'm still scratching my head on how to add PID (and maybe TID if it's important) to the WorkUnitMetadata so the entries aren't overlapped

I tried adding it in Rustland, but it's always the same PID (which probably makes sense... but I don't know why 😉 )

Chrome Trace JSON is pretty powerful. Right now I'm just using the simple data format, which understands PIDs and TIDs, and will render them in different swimlanes

If possible, I'd like to make the change(s) to Pants myself to help me understand the engine/architecture better. So feel free to reach out and I can pester whoever with questions 😛

I'll keep goofing off a bit more and then put it on the shelf

I started with

speedscope

but that doesn't support multi-process

@hundreds-father-404 I was a C++ "guru" 🤮 at my last company a few years ago so Rust is too far off from what I already know. I also know it's a better C++ and advocated that we switch, even without "knowing" it 😂

Chrome Traces also support more interesting use-case pother than event spans (nested start+end) like asynchronous events (which I've never used). So if you wanted to get really detailed, you could model Pants' asynchronous behavior pretty closely (have the resulting graph possibly show tasks that were started/stopped)

but really glad that you’re looking at this! one useful avenue is likely to be looking at “self time” vs “cumulative time” for workunits

it’s the next thing i was planning on exploring for overall performance.

the only thing that is likely to be tricky is that because workunits form a tree rather than a dag (even though the underlying structure is a dag), when the “second parent” of a node in the graph adds a dependency on something that is already running, it won’t actually end up with a child (because it wouldn’t form a tree). unfortunately that means that that “second parent” would end up with a bunch of false self-time: it’s blocked waiting for a node, but that’s not recorded in the workunit tree.

having said that: the reason that they form a tree is mostly in order to support profilers like Zipkin and Chrome. and it would be easy enough to transform a DAG into a tree. so maybe the best solution would be to migrate to actually storing workunits (and giving them to WorkunitsCallbacks) as DAGs, and to then let whomever wants to render in Zipkin/Chrome transform to a tree.

or, if you’re interested in the time taken to do cache lookups or sandbox setup, can filter to all children of

multi_platform_process

nodes.

those two are recorded on the rust side like so: https://github.com/pantsbuild/pants/blob/355d873899f31165a4e2ff1b84bc9aa3fb1c5575/src/rust/engine/process_execution/src/local.rs#L258-L268

But it sounds like something like this is on the radar, and obviously y'all have a much deeper and contextual understanding of the code and possible solution(s) 😉

at least for your purposes, the fact that we lie and treat the DAG as a tree is helpful (although a bit misleading)

The

@rule(desc=...)

and rule function name end up in workunits.