Hey everyone once again im having performance issues with th

Question

Hey everyone once again im having performance issues with the v2 engine and looking for suggestions Here is my setup ```circleci 8 CPU 16 RAM``` I am trying to run tests via pytest or behave In this environment we start up our servers and run the tests we have about 8 webservers running with pants and we kick off the tests I am getting out of memory errors and process killed issues when running If you have been following what i have been posting about i am having all sorts of performance issues I am running pants 2 0 1rc4 with a twist from a release that was cut for me yesterday There is no way the performance of v2 can be this much worse than v1 as a whole At this point im thinking im missing or overlooking something Any suggestions

hundreds-father-404 · Answer

I haven t been following the past threads closely apologies if you ve already answered Do you know where the costs are mostly coming from Some candidates Cache upload and download Resolving requirements Pants overhead when setting up tests The test processes themselves that they re slower than they were before Running with ` no dynamic ui` may be instructive as it will output the start time and end time for things

witty-crayon-22786 · Answer

<https pantsbuild slack com archives C046T6T9U p1608149983219400> is a previous thread on this

witty-crayon-22786 · Answer

my understanding is that you are spawning N ` pants run` clients which are hitting M ` pants run` servers on the same box

witty-crayon-22786 · Answer

is that right

salmon-barista-63163 · Answer

yes < witty crayon 22786> but that part has been resolved now throwing one more pants process to run tests or execute pytests on top of that really makes things go crazy

salmon-barista-63163 · Answer

little more context cci pants cache is loaded no issue the dynamic ui is off pantsd is off Here are the things that are for sure slower with pants v2 than v1 pants startup processes pants building a pex file running pytest running our behave tests

witty-crayon-22786 · Answer

so during the portion of ` pants run` where the process is spawned `pants` itself should be using effectively no CPU at all but it is definitely using memory this was why john and i were curious about the OOM killer

witty-crayon-22786 · Answer

gt Here are the things that are for sure slower with pants v2 than v1 when `pantsd` is off it is not surprising that the first item is slower `pantsd` is intended to cut off the startup time

witty-crayon-22786 · Answer

and all other tasks will be lightly affected as well

witty-crayon-22786 · Answer

we re working on optimizing `pytest` runs because there is some extra overhead there that we know about

witty-crayon-22786 · Answer

but i think that that might all be moot if your processes are getting killed i am only aware of the OOM killer being able to do that in linux i do not know of a facility to kill things based on CPU usage

salmon-barista-63163 · Answer

I wasnt aware of anything that would kill processes for CPU either but if i watch the avail memory there is plenty there when it happens

witty-crayon-22786 · Answer

are the clients running `pytest` or some other code

salmon-barista-63163 · Answer

its pytest

witty-crayon-22786 · Answer

and this can t be phrased as a single client running a single run of pants running multiple tests concurrently

salmon-barista-63163 · Answer

i get alot of this error in my pytest ``` Error reading file File path coverage is executable false Os code 2 kind NotFound message No such file or directory ``` which typically happens when there is no memory left to finish writing the coverage report

salmon-barista-63163 · Answer

im about to turn off the coverage feeatures and see if this helps

salmon-barista-63163 · Answer

If i bump the circle ci executer size up one things start to smooth out but this isnt something i can do long term as they are very expensive

witty-crayon-22786 · Answer

at a fundamental level changing your client to be a single run of ` pants test $multiple targets` with parallelism enabled is what would be ideal as that would use dramatically less memory

witty-crayon-22786 · Answer

having tons of independent instances of pants is just not the intended usage right now we ll keep working on <https github com pantsbuild pants issues 7654> but it ll be at least a few weeks

salmon-barista-63163 · Answer

so for the tests its one ` pants test` execituon

salmon-barista-63163 · Answer

im struggling with performance impacts everywhere we run pants in our repo

witty-crayon-22786 · Answer

is that after the ` pants run` clients have completed or concurrently

salmon-barista-63163 · Answer

so when tests run its going to call into our webservers that are running with pants to me this looks like too many running pants processes at once where with v1 we didnt run into this

salmon-barista-63163 · Answer

it is after the pants run cleints are up and running though

witty-crayon-22786 · Answer

up and running or done and exited

salmon-barista-63163 · Answer

up and running

witty-crayon-22786 · Answer

ok so you have ~17 pants processes at once

salmon-barista-63163 · Answer

we could have upwards of 20+ yes

witty-crayon-22786 · Answer

v2 uses more memory to a degree that is intended and expected it s keeping work warm for further runs

witty-crayon-22786 · Answer

which you can only take advantage of with `pantsd`

hundreds-father-404 · Answer

I think you mentioned trying running ` pants package` followed by `dist app pex` rather than ` pants run` Iirc you stopped that approach because it was still too much contention when building the PEXes Now that you have the building of PEXes part fixed might it be worth trying that approach again Then you have ~20 Pex processes running and only 1 Pants process for Pytest

witty-crayon-22786 · Answer

so i think that as discussed yesterday i think you either need to 1 switch to ` pants package amp amp dist app pex` 2 wait for <https github com pantsbuild pants issues 7654> and we could try to prioritize it

salmon-barista-63163 · Answer

I am running ` pants package amp amp dist app pex` for the one webserver that would spin up tasks pants processes on demand We pre package ~30 pexs and then execute by just running the pex This is the rest of our stack that is causing issues now Its just too many things running in pants i guess

hundreds-father-404 · Answer

It sounds like you ve been playing with this already but another thing you could continue tweaking is ` process execution local parallelism` Note that you can use it as a CLI option not only a config file value So you can set it to a higher value when building PEXes then lower it dramatically when running tests In v1 it was effectively set to 1 Tests ran sequentially

hundreds-father-404 · Answer

Hm possibly worth trying to see the impact ` pants test debug ` emulates the v1 behavior It runs each test sequentially in the foreground

witty-crayon-22786 · Answer

Screen Shot 2020 12 18 at 10 07 57 png

witty-crayon-22786 · Answer

witty-crayon-22786 · Answer

basically it looks like circle have a memory usage killer other than the kernel

salmon-barista-63163 · Answer

hmmm

salmon-barista-63163 · Answer

i have never seen that file

witty-crayon-22786 · Answer

it s an older post newer posts about memory limits have more information

salmon-barista-63163 · Answer

I have tried running tests sequentially like in v1 by setting the process execution local parallelism to 1 same issues

salmon-barista-63163 · Answer

ill check into some of then circle ci memory posts

witty-crayon-22786 · Answer

witty-crayon-22786 · Answer

but basically if you try to search for CPU usage killer for circleci you only find information about memory usage

salmon-barista-63163 · Answer

yeah i know the exit code 137 oom one I litterally get a message from my linux container that says ```Killed lt pid gt Out of memory``` thats what i am fighting in some test jobs here and others just all the sudden it exits with exit code 1

salmon-barista-63163 · Answer

the placese where it kills itself is always different never consistent

witty-crayon-22786 · Answer

yea ok <https pantsbuild slack com archives C046T6T9U p1608314670274500 thread ts=1608308897 266800 amp cid=C046T6T9U> is the way forward then i think

salmon-barista-63163 · Answer

okay thanks for chatting regarding it i was just trying to get any last ditch efforts to try to solve this

witty-crayon-22786 · Answer

the effect of that would be significantly reducing the amount of pants runs running at once

salmon-barista-63163 · Answer

looks like ill have to wait for this

salmon-barista-63163 · Answer

do you have an estimated ETA for that ticket to complete into a future release I would like to discuss this with my team

witty-crayon-22786 · Answer

the ` pants binary package amp amp dist app pex` change should work in either v1 or v2 are you sure that that isn t an option

salmon-barista-63163 · Answer

yes this works but we cannot use that everywhere due to the nature of our system and architecture

salmon-barista-63163 · Answer

I implemented it everywhere i could

witty-crayon-22786 · Answer

the ~8 clients and ~8 servers are the critical spot i think

salmon-barista-63163 · Answer

yeah that seems to be what i am seeing as well

witty-crayon-22786 · Answer

<https pantsbuild slack com archives C046T6T9U p1608315718280000 thread ts=1608308897 266800 amp cid=C046T6T9U> i ll get you an answer to this in about 2 hours

witty-crayon-22786 · Answer

BUT unfortunately we definitely won t be able to backport that change to 2 0 x 2 1 x it s built atop a bunch of stuff that only exists in 2 2 x

salmon-barista-63163 · Answer

okay thank you

witty-crayon-22786 · Answer

so i think that we can commit to having this done in January

salmon-barista-63163 · Answer

so we are looking late jan release

witty-crayon-22786 · Answer

yea done and shipped in January

salmon-barista-63163 · Answer

ok thank you < witty crayon 22786>

salmon-barista-63163 · Answer

I will chat with ym team but looks like we are going to halt our rollout of pants 2 until this happens

witty-crayon-22786 · Answer

i m a broken record but i do think that the binary package approach would work around the issue in this case but we ll be prioritizing the pantsd change because i 100 agree that it shouldn t be necessary to avoid concurrent runs

happy-kitchen-89482 · Answer

So we can get some context can you elaborate on the reason that in some cases you have to run a server with ` pants run lt tgt gt ` and cannot switch to ` pants package lt tgt gt amp amp dist lt tgt gt pex`

happy-kitchen-89482 · Answer

We d like to understand the use case better

happy-kitchen-89482 · Answer

Thanks

salmon-barista-63163 · Answer

< happy kitchen 89482> So we have a server lets call it the parent that runs via a pants run process That server calls subprocesses that will run a task Lets call those child services Those tasks are executed with a pants run command We have since migratred those tasks to use a pants package and then run the pex subprocess The issue here is that we can spin up ~30 of tasks which gives us ~30 pants package processes running in parallel I hope this answers the question as its kind of hard to explain what we use this server for without explaining alot of our product infrastructure

salmon-barista-63163 · Answer

I do have a question Will the fix for pantsd be backported to pants 2 1 or will it only exist in 2 2 We are on 2 0 currently just trying to see what I need to update to in preparation for this release of the bug fix

hundreds-father-404 · Answer

< witty crayon 22786> to confirm we would not backport pantsd support for concurrent runs because it is impossible to backport without breaking the deprecation policy right

hundreds-father-404 · Answer

gt The issue here is that we can spin up ~30 of tasks which gives us ~30 pants package processes running in parallel < salmon barista 63163> to clarify it sounds like it is not possible to invoke those all in a single run Like ` pants package tgt1 tgt2 tgt3`

salmon-barista-63163 · Answer

correct < hundreds father 404> we do this on the fly as tasks spin up with our application lwe cannot do all the processes in a single run unfortunately

witty-crayon-22786 · Answer

no it would just be a backport of something like 6 or 7 patches it would be a big investment to ensure that it was stable

hundreds-father-404 · Answer

I thought that we must remove all global state to land this change Meaning `Subsystem global instance ` We can t backport that because the deprecation policy