pantsbuild #development

I only mention it because I’m fairly sure multiprocessing.pool is just bad and have had success in past projects with zmq for hardcore multiprocess utilization

rough-minister-58256

08/11/2016, 11:33 PM

(iff we get to multiprocess feasibility)

fancy-queen-20734

08/12/2016, 12:23 AM

Anyone else want to look at : https://rbcommons.com/s/twitter/r/4150/? It’s pretty small

fancy-queen-20734

08/12/2016, 12:35 AM

going to submit it.

rough-minister-58256

08/12/2016, 12:36 AM

#shipit

rough-minister-58256

08/12/2016, 12:39 AM

@fancy-queen-20734: do the updates in https://reviewboard.twitter.biz/r/861199/ look good to you?

fancy-queen-20734

08/12/2016, 12:40 AM

shipit!

rough-minister-58256

08/12/2016, 12:40 AM

👍🏼

witty-crayon-22786

08/12/2016, 12:43 AM

looks good @fancy-queen-20734

rough-minister-58256

08/12/2016, 12:45 AM

@salmon-summer-38098: were you gonna land https://rbcommons.com/s/twitter/r/4137/ soon too? I think after that we should be good to roll another pex and sandbox w/ high probability of a green run.

rough-minister-58256

08/12/2016, 12:46 AM

then we just need a source-pants release w/ our changes + some pants-plugins integration tests to seal this puppy into master

salmon-summer-38098

08/12/2016, 1:56 AM

@rough-minister-58256: yes, hope to get to it tonight.

salmon-summer-38098

08/12/2016, 1:57 AM

otherwise, tomorrow for sure!

rough-minister-58256

08/12/2016, 2:01 AM

awesome thx!

billowy-beach-68498

08/12/2016, 2:33 AM

@fancy-queen-20734: regarding https://rbcommons.com/s/twitter/r/4150/, this change only affects v2 engine right? because with v1 engine, fileset.rel_root is still BUILD dir, thus will end up getting ValueError if rel_path is given (unless rel_path happens to be BUILD dir)

rough-minister-58256

08/12/2016, 6:15 AM

noticed through some various testing today that the daemon felt slower than expected - I think it’s a symptom of forking of a 7+GB process

rough-minister-58256

08/12/2016, 6:17 AM

also noticed that we seem to have a memory leak of some sort - repeated runs of

./pants list ::

with the daemon add on .5GB per run, which leads to daemon getting slower each run

rough-minister-58256

08/12/2016, 6:25 AM

appears that roughly 20% of the time of a daemon

./pants list ::

is spent just forking

🐌 2

billowy-beach-68498

08/12/2016, 2:28 PM

@rough-minister-58256: I think memory leak is likely. at first i thought python has its own garbage collection thus it’s impossible to forget to do “free” like in C. but if a variable never goes out of scope and is never used, then it is in fact a memory leak.

fancy-queen-20734

08/12/2016, 3:48 PM

@billowy-beach-68498: it should only affect the v2 engine. The validation fails if any glob is passed, except the v2 variant. But, if a glob were constructed that looked like the v2’s usage it would also work and that would be ok.

billowy-beach-68498

08/12/2016, 4:20 PM

anyone opposing to change Noop('Edge would cause a cycle: {} -> {}.', src, dst) to Noop(‘Cycle detected! Edge would ……’)? an integration test is looking for “Cycle detected” string, and it is used in v1 engine.

rough-minister-58256

08/12/2016, 5:26 PM

works for me

rough-minister-58256

08/12/2016, 5:26 PM

and yeah, python will only garbage collect things when there are no more references to them

rough-minister-58256

08/12/2016, 5:28 PM

seems like we might want to actually set a ttl on nodes in the graph in some cases to auto-invalidate after some time