#development
Title
e
early-needle-54791
11/27/2019, 9:55 PMIs there any parallelism in the v1 nailgun executor when compiling java/scala code or is only one client connected to the server at a time?
a
aloof-angle-91616
11/27/2019, 9:56 PMthe v1
NailgunTask--worker-countso there is lots of parallelism there, and i believe we actually have a single
NailgunClientNailgunClientSessionpants will create literal python threads equal to the value of
--worker-counthttps://github.com/pantsbuild/pants/pull/6579 this is a really really old PR that attempted to make this more explicit by using contextmanagers to represent nailgun client/session state
so basically there is parallelism, which we get by spinning up multiple python worker threads. python threads can only execute one at a time, but since each thread basically just blocks on i/o and sends i/o chunks back to pants, we can get lots of parallelism because python threads can make use of parallelism if there's a lot of i/o
let me know if that makes sense or not, we can pair too
e
early-needle-54791
11/27/2019, 10:03 PMthank you! makes sense.
a
aloof-angle-91616
11/27/2019, 10:03 PMlit
note that one of the reasons why this implementation is used right now is that it allows
NailgunTask--execution-strategy=subprocessbecause with
--execution-strategy=subprocessso like, that's a useful facet of the current architecture, because it allowed for the generic
--execution-strategybut that's just to explain why it is the way it is -- we have different constraints now, so no need to copy that directly
e
early-needle-54791
12/02/2019, 6:49 PMBuilding on this. I believe v1 starts multiple nailgun servers. is that correct?
a
aloof-angle-91616
12/02/2019, 7:04 PMfor a single jvm tool, v1 starts a single nailgun server process with the zinc classpath. it will then create N threads in python, each of which will:
1. take a compile request (for a single target) off the queue
2. create a nailgun request
3. connect to the single nailgun server with that request
4. write to the cache, etc when the request is complete
in this case, the multithreading is done in python. we could simulate this in rust if we:
1. create a single nailgun server process in the background
2. create a future for each target which creates a nailgun request (blocking until a target is available to compile)
3. in each future, connect to the single nailgun server, and execute the request, “blocking” on its output
let me know if that makes sense!
just edited the above message to add step 1 in rust (create the single server process)
e
early-needle-54791
12/02/2019, 7:06 PMyeah it does. I’m doing multithreading with tokio now, but running in to some race condition which I am having a hard time sorting out.
a
aloof-angle-91616
12/02/2019, 7:07 PMoh that’s not fun
e
early-needle-54791
12/02/2019, 7:08 PMI talked to @witty-crayon-22786 a few times about increasing parallelism in hermetic nailgun, and we landed on 1 server per client, and not multithreading. I can’t remember why that was now tho…should have written a 1 pager.
a
aloof-angle-91616
12/02/2019, 7:08 PMit’s a heck of a lot easier to implement i think
e
early-needle-54791
12/02/2019, 7:09 PMYeah not fun, but I was mostly trying to remember if there reason it wasn’t working was because our nailgun had some issues that made threading hard.
a
aloof-angle-91616
12/02/2019, 7:09 PMdo you want to pair on this? i can probably screenshare for a bit. can’t guarantee we’ll magically fix the race though
e
early-needle-54791
12/02/2019, 7:10 PMNot right now, I’m going to get something reviewable without parallelism before I pick it back up.
a
aloof-angle-91616
12/02/2019, 7:10 PMi think one thing that’s relevant is that the compiles have to be done in dependency order. in python you can just pop things off a queue and i think the GIL may make it easier than in rust where you can actually have threading
that sounds like a great idea
e
early-needle-54791
12/02/2019, 7:11 PMYeah, but I assumed that the node runner in v2 was dealing with dependency order. Like it wouldn’t schedule a node until all the futures for its dependencies had finished.
a
aloof-angle-91616
12/02/2019, 7:11 PMfeel free to add me as reviewer, i’m trying to get back into that (no sweat though)
yeah that sounds right actually
but that being said, that would be true if we had compile 100% in v2, which we don’t quite yet
e
early-needle-54791
12/02/2019, 7:12 PMAh interesting point
a
aloof-angle-91616
12/02/2019, 7:12 PMbut i think that controlling it from python might get the same result
e
early-needle-54791
12/02/2019, 7:12 PMthat could be the crux
I will investigate more later. Thank!
a
aloof-angle-91616
12/02/2019, 7:12 PMyou’re welcome!
e
early-needle-54791
12/02/2019, 7:13 PMbecause now we are only scheduling 1 compile request at a time with product_request or some such thing?
👍 1
a
aloof-angle-91616
12/02/2019, 7:15 PMyes that’s it