pantsbuild #development

Channels

welcome

github-notifications

development

witty-crayon-22786

09/27/2019, 12:55 AM

processes are where we have our escape hatches for that

witty-crayon-22786

09/27/2019, 12:55 AM

it's not about synchronous vs asynchronous so much as about being reproducible/sideeffect free

aloof-angle-91616

09/27/2019, 12:55 AM

process executions are persistently cacheable for free

hundreds-breakfast-49010

09/27/2019, 12:56 AM

when you say process are you talking about spawning a different os thread, or is this something different?

aloof-angle-91616

09/27/2019, 12:56 AM

it might be interesting to avoid the reliance on that particular mechanic by expanding the kinds of operations that can be persistently cached

average-vr-56795

09/27/2019, 10:09 AM

@hundreds-breakfast-49010 We currently have a

list-and-die-for-testing

goal in

pants-plugins/src/python/internal_backend/rules_for_testing/register.py

- you could add another there which does materialisation

average-vr-56795

09/27/2019, 10:10 AM

My wild guess about that commit you pointed at is that you’re adding a list and two functions, and that wrapping

optionable_rule(StubOutput) + stub_rule

in a list would maybe help, but I don’t massively know what’s going on from the stack trace 🙂

happy-kitchen-89482

09/27/2019, 2:38 PM

Hi all, I've started taking a look at supporting "named rules" or "traceable rules" or "public rules" (or whatever we want to call them) - basically the rules that are displayed in the UI, including Zipkin traces, as opposed to "anonymous rules" or "private rules" or (whatever we want to call them) that are implementation details and not of interest to the end user.

happy-kitchen-89482

09/27/2019, 2:38 PM

We talked about this in person a couple of weeks ago if you recall.

happy-kitchen-89482

09/27/2019, 2:39 PM

Anyway, I could use a little handholding around understanding the WorkUnitStore, which appears to be the salient concept.

happy-kitchen-89482

09/27/2019, 2:40 PM

[PS I have a zipkin server running and can post v2 engine traces to it and view them in the UI]

happy-kitchen-89482

09/27/2019, 2:41 PM

As far as I can tell, the only things that get zipkin-traced on the v2 side are WorkUnits in the WorkUnitStore: https://github.com/pantsbuild/pants/blob/fe3d9834c0e2d78bb252604d6572ffd3b7ac5226/src/rust/engine/engine_cffi/src/lib.rs#L372

happy-kitchen-89482

09/27/2019, 2:43 PM

What I'm trying to make sense of is the calls to

add_work_unit()

happy-kitchen-89482

09/27/2019, 2:45 PM

I see some in

src/rust/engine/fs/store/src/remote.rs

and in

src/rust/engine/process_execution/src/remote.rs

(what's the difference between those two files btw?) I naively assume those are all to do with tracing remote execution?

happy-kitchen-89482

09/27/2019, 2:46 PM

Then there's one in

<http://nodes.rs|nodes.rs>

, which I have to assume is tracing every rule?

happy-kitchen-89482

09/27/2019, 2:46 PM

Am I vaguely on track so far?

average-vr-56795

09/27/2019, 2:51 PM

@dry-policeman-7927 ^^

average-vr-56795

09/27/2019, 2:55 PM

From the top of my head: There is a flag to enable tracing every node that is executed in v2, as well as every file upload/download (in the fs/store/src/remote.rs) and every execution request (in process_execution/src/remote.rs), as well as metadata about remote-side timings that the execution server may have reported (e.g. in the scoot implementation, it reports time spent in remote queues, time spent for the worker to fetch input files, time spent for the worker to upload output files, etc)

average-vr-56795

09/27/2019, 2:55 PM

The v2 spans don’t currently have proper parent-child relationships between them; help welcome!

average-vr-56795

09/27/2019, 2:57 PM

Otherwise, sounds about right… Any specific ideas/problems/questions?

happy-kitchen-89482

09/27/2019, 3:24 PM

That's very helpful

happy-kitchen-89482

09/27/2019, 3:25 PM

So the rule timings themselves are reported in

<http://nodes.rs|nodes.rs>

happy-kitchen-89482

09/27/2019, 3:41 PM

Also, are these WorkUnits used for anything other than zipkin reporting at the moment?

aloof-angle-91616

09/27/2019, 3:42 PM

i'm hacking on that since it's actually not too hard to get the

RunTracker

as a v2 engine param and start spawning workunits to write to

aloof-angle-91616

09/27/2019, 3:42 PM

will update after demo

happy-kitchen-89482

09/27/2019, 3:42 PM

And finally, can you elaborate on the parent-child issues? In general rules form a DAG not a tree, so there may not be a single "parent", although in practice I guess the first rule to trigger a rule would be the acting parent of that triggered rule.

dry-policeman-7927

09/27/2019, 3:43 PM

The rule timing is a part of WorkUnit. WorkUnit are used for Zipkin tracing in Rust part

aloof-angle-91616

09/27/2019, 3:43 PM

i think that the use of the word "parent" can be misleading here, a "back pointer" might be more appropriate?

👍 1

dry-policeman-7927

09/27/2019, 3:46 PM

Copy code

in practice I guess the first rule to trigger a rule would be the acting parent of that triggered rule

I think this is how we want to view parent-child relation for v2, but I @average-vr-56795 and @witty-crayon-22786 may correct me or give more context

happy-kitchen-89482

09/27/2019, 3:48 PM

Danny, what are you hacking on exactly? My goal is to have an uploaded report from v2 similar to the one we already have from the old engine's RunTracker. Is that what you're already doing?