Hello So I am in the process of moving our main repo over to

Question

Hello So I am in the process of moving our main repo over to pants 2 Our dev environment that is run for each developer runs as follows centOS ec2 instance that runs a set of docker containers each docker container uses the same base image Pants along with all of thee code is bind mounted into the containers the entrypoint and command for each docker container is a ` pants run` command This all worked great with pants v1 and we mounted in the cache so everything was quick and worked well With pants 2 if i start up one container everything works just fine when we start up more than 1 container i get the following error ```config v1 | 22 06 54 17 ERROR Exception caught pants engine internals scheduler ExecutionError config v1 | File home centos cache pants setup bootstrap Linux x86 64 2 0 2 py36 lib python3 6 site packages pants bin local pants runner py line 289 in run config v1 | engine result = self run v2 config v1 | File home centos cache pants setup bootstrap Linux x86 64 2 0 2 py36 lib python3 6 site packages pants bin local pants runner py line 195 in run v2 config v1 | return self maybe run v2 body goals poll=False config v1 | File home centos cache pants setup bootstrap Linux x86 64 2 0 2 py36 lib python3 6 site packages pants bin local pants runner py line 217 in maybe run v2 body config v1 | poll delay= 0 1 if poll else None config v1 | File home centos cache pants setup bootstrap Linux x86 64 2 0 2 py36 lib python3 6 site packages pants init engine initializer py line 127 in run goal rules config v1 | goal product params poll=poll poll delay=poll delay config v1 | File home centos cache pants setup bootstrap Linux x86 64 2 0 2 py36 lib python3 6 site packages pants engine internals scheduler py line 569 in run goal rule config v1 | self raise on error t for t in throws config v1 | File home centos cache pants setup bootstrap Linux x86 64 2 0 2 py36 lib python3 6 site packages pants engine internals scheduler py line 539 in raise on error config v1 | wrapped exceptions=tuple t exc for t in throws config v1 | config v1 | Exception message 1 Exception encountered config v1 | config v1 | Exception Failed to begin read transaction Resource temporarily unavailable``` I suspect this has something to do with how we are mounting in the cache dirs IE we mount `~ cache pants gt ~ cache pants` If we do not mount in the cache all the services come up no problem but each has to build and resolve all the constraints first which takes a long time Any thoughts here Am I missing something Any suggestions on what to try to resolve this

happy-kitchen-89482 · Answer

Hmmm what happens if you run without pantsd ` no pantsd`

salmon-barista-63163 · Answer

This is without pantsd < happy kitchen 89482>

happy-kitchen-89482 · Answer

Huh ccing < witty crayon 22786> for later after his build meetup talk

salmon-barista-63163 · Answer

Yeah if i have pantsd FALSE and concurrent TRUE this starts to happen with pantsd TRUE and concurrent FALSE I get errors like this ```config v2 | 16 24 24 10 INFO initializing pantsd mock stitch api exited with code 1 config v2 | 16 24 25 08 INFO pantsd initialized config v2 | Exception in thread SchedulerServiceThread config v2 | Traceback most recent call last config v2 | File home centos pyenv versions 3 6 12 lib python3 6 threading py line 916 in bootstrap inner config v2 | self run config v2 | File home centos pyenv versions 3 6 12 lib python3 6 threading py line 864 in run config v2 | self target self args self kwargs config v2 | File home centos cache pants setup bootstrap Linux x86 64 2 0 2 py36 lib python3 6 site packages pants pantsd service pants service py line 219 in target config v2 | service run config v2 | File home centos cache pants setup bootstrap Linux x86 64 2 0 2 py36 lib python3 6 site packages pants pantsd service scheduler service py line 141 in run config v2 | self invalidation globs and snapshot = globs self get snapshot globs poll=False config v2 | File home centos cache pants setup bootstrap Linux x86 64 2 0 2 py36 lib python3 6 site packages pants pantsd service scheduler service py line 85 in get snapshot config v2 | timeout=timeout config v2 | File home centos cache pants setup bootstrap Linux x86 64 2 0 2 py36 lib python3 6 site packages pants engine internals scheduler py line 593 in product request config v2 | self raise on error t for t in throws config v2 | File home centos cache pants setup bootstrap Linux x86 64 2 0 2 py36 lib python3 6 site packages pants engine internals scheduler py line 539 in raise on error config v2 | wrapped exceptions=tuple t exc for t in throws config v2 | pants engine internals scheduler ExecutionError 1 Exception encountered config v2 | config v2 | Exception Failed to begin read transaction Resource temporarily unavailable config v2 | config v2 | config v2 | 16 24 25 23 ERROR Exception caught pants engine internals scheduler ExecutionError config v2 | File home centos cache pants setup bootstrap Linux x86 64 2 0 2 py36 lib python3 6 site packages pants bin local pants runner py line 289 in run config v2 | engine result = self run v2 config v2 | File home centos cache pants setup bootstrap Linux x86 64 2 0 2 py36 lib python3 6 site packages pants bin local pants runner py line 195 in run v2 config v2 | return self maybe run v2 body goals poll=False config v2 | File home centos cache pants setup bootstrap Linux x86 64 2 0 2 py36 lib python3 6 site packages pants bin local pants runner py line 217 in maybe run v2 body config v2 | poll delay= 0 1 if poll else None config v2 | File home centos cache pants setup bootstrap Linux x86 64 2 0 2 py36 lib python3 6 site packages pants init engine initializer py line 127 in run goal rules config v2 | goal product params poll=poll poll delay=poll delay config v2 | File home centos cache pants setup bootstrap Linux x86 64 2 0 2 py36 lib python3 6 site packages pants engine internals scheduler py line 569 in run goal rule config v2 | self raise on error t for t in throws config v2 | File home centos cache pants setup bootstrap Linux x86 64 2 0 2 py36 lib python3 6 site packages pants engine internals scheduler py line 539 in raise on error config v2 | wrapped exceptions=tuple t exc for t in throws config v2 | config v2 | Exception message 1 Exception encountered config v2 | config v2 | Exception Failed to begin read transaction Resource temporarily unavailable config v2 | config v2 | config v2 | Use print stacktrace to see more error details config v2 | PID 61 waiting for a file lock ~ pids lock pantsd held by 60 home centos cache pants setup bootstrap Linux x86 64 2 0 2 py36 bin python home centos cache pants setup bootstrap Linux x86 64 2 0 2 py36 bin pants pants bin name= pants pants version=2 0 2 run servers api runtime src runtime server config v2 exited with code 1```

salmon-barista-63163 · Answer

It almost looks like each container is trying to spin up its own instance of the daemon ideally i want each container to share the daemon on the host and then share the cache via binded mounts theres got to be some setting or something im missing here

salmon-barista-63163 · Answer

heres my toml file maybe something will jump out ``` GLOBAL level = info pants version = 2 0 2 colors = true pantsd = true concurrent = false Run pants invocations and processes in parallel process execution local parallelism = 4 Process Directories pants subprocessdir = ~ pids source root patterns = src python setup resolver jobs = 2 interpreter constraints = CPython gt =3 6 lt 3 7 requirement constraints = 3rdparty python constraints txt interpreter search paths = lt PYENV gt lt PATH gt python repos repos = buildroot s 3rdparty python repo ```

hundreds-father-404 · Answer

I see `pantsd = true` and `concurrent = false` Are you disabling Pantsd another way

salmon-barista-63163 · Answer

ah sorry i was messing with the settings trying it multiple ways

salmon-barista-63163 · Answer

Let me ask this With the docker architecture we have where we run a single service with `pants run` in a container how should i set these options What is your opinion here for best performance

witty-crayon-22786 · Answer

tied up for a few hours but then will respond here sorry Brian

witty-crayon-22786 · Answer

so workflows involving running lots of copies of pants are just not advisable until we ve resolved <https github com pantsbuild pants issues 7654> i just got back from paternity leave and will be able to resume it in the coming weeks once the other blocker is resolved

witty-crayon-22786 · Answer

but even then if pants is running in a bunch of containers they ll never be able to share resources across instances the way that `pantsd` depends on for performance

witty-crayon-22786 · Answer

i think we ve previously discussed using ` pants package` followed by running the package rather than ` pants run` and i think that i would continue to recommend that for the i need to run X binaries concurrently case you d disable `pantsd` inside the container as well

witty-crayon-22786 · Answer

if they weren t in separate docker containers then <https github com pantsbuild pants issues 7654> would be helpful because it would allow all of the resources to be shared across processes but inside the container there is no sharing