https://pantsbuild.org/ logo
#general
Title
# general
s

salmon-barista-63163

06/24/2021, 10:18 PM
Hello. So I am in the process of moving our main repo over to pants 2. Our “dev” environment that is run for each developer runs as follows. • centOS ec2 instance that runs a set of docker containers • each docker container uses the same base image • Pants (along with all of thee code) is bind mounted into the containers • the entrypoint and command for each docker container is a
./pants run
command. This all worked great with pants v1 and we mounted in the cache so everything was quick and worked well. With pants 2 if i start up one container everything works just fine, when we start up more than 1 container i get the following error.
Copy code
config-v1                           | 22:06:54.17 [ERROR] Exception caught: (pants.engine.internals.scheduler.ExecutionError)
config-v1                           |   File "/home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.2_py36/lib/python3.6/site-packages/pants/bin/local_pants_runner.py", line 289, in run
config-v1                           |     engine_result = self._run_v2()
config-v1                           |   File "/home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.2_py36/lib/python3.6/site-packages/pants/bin/local_pants_runner.py", line 195, in _run_v2
config-v1                           |     return self._maybe_run_v2_body(goals, poll=False)
config-v1                           |   File "/home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.2_py36/lib/python3.6/site-packages/pants/bin/local_pants_runner.py", line 217, in _maybe_run_v2_body
config-v1                           |     poll_delay=(0.1 if poll else None),
config-v1                           |   File "/home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.2_py36/lib/python3.6/site-packages/pants/init/engine_initializer.py", line 127, in run_goal_rules
config-v1                           |     goal_product, params, poll=poll, poll_delay=poll_delay
config-v1                           |   File "/home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.2_py36/lib/python3.6/site-packages/pants/engine/internals/scheduler.py", line 569, in run_goal_rule
config-v1                           |     self._raise_on_error([t for _, t in throws])
config-v1                           |   File "/home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.2_py36/lib/python3.6/site-packages/pants/engine/internals/scheduler.py", line 539, in _raise_on_error
config-v1                           |     wrapped_exceptions=tuple(t.exc for t in throws),
config-v1                           |
config-v1                           | Exception message: 1 Exception encountered:
config-v1                           |
config-v1                           |   Exception: Failed to begin read transaction: Resource temporarily unavailable
I suspect this has something to do with how we are mounting in the cache dirs: IE we mount
~/.cache/pants -> ~/.cache/pants
If we do not mount in the cache all the services come up no problem but each has to build and resolve all the constraints first which takes a long time. Any thoughts here? Am I missing something? Any suggestions on what to try to resolve this?
h

happy-kitchen-89482

06/25/2021, 1:03 AM
Hmmm, what happens if you run without pantsd (
--no-pantsd
) ?
s

salmon-barista-63163

06/25/2021, 12:04 PM
This is without pantsd @happy-kitchen-89482
h

happy-kitchen-89482

06/25/2021, 4:26 PM
Huh. ccing @witty-crayon-22786 for later (after his build meetup talk)
s

salmon-barista-63163

06/25/2021, 4:30 PM
Yeah if i have pantsd FALSE and concurrent TRUE this starts to happen. with pantsd TRUE and concurrent FALSE I get errors like this:
Copy code
config-v2                           | 16:24:24.10 [INFO] initializing pantsd...
mock-stitch-api exited with code 1
config-v2                           | 16:24:25.08 [INFO] pantsd initialized.
config-v2                           | Exception in thread SchedulerServiceThread:
config-v2                           | Traceback (most recent call last):
config-v2                           |   File "/home/centos/.pyenv/versions/3.6.12/lib/python3.6/threading.py", line 916, in _bootstrap_inner
config-v2                           |     self.run()
config-v2                           |   File "/home/centos/.pyenv/versions/3.6.12/lib/python3.6/threading.py", line 864, in run
config-v2                           |     self._target(*self._args, **self._kwargs)
config-v2                           |   File "/home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.2_py36/lib/python3.6/site-packages/pants/pantsd/service/pants_service.py", line 219, in target
config-v2                           |     service.run()
config-v2                           |   File "/home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.2_py36/lib/python3.6/site-packages/pants/pantsd/service/scheduler_service.py", line 141, in run
config-v2                           |     self._invalidation_globs_and_snapshot = (globs, self._get_snapshot(globs, poll=False))
config-v2                           |   File "/home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.2_py36/lib/python3.6/site-packages/pants/pantsd/service/scheduler_service.py", line 85, in _get_snapshot
config-v2                           |     timeout=timeout,
config-v2                           |   File "/home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.2_py36/lib/python3.6/site-packages/pants/engine/internals/scheduler.py", line 593, in product_request
config-v2                           |     self._raise_on_error([t for _, t in throws])
config-v2                           |   File "/home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.2_py36/lib/python3.6/site-packages/pants/engine/internals/scheduler.py", line 539, in _raise_on_error
config-v2                           |     wrapped_exceptions=tuple(t.exc for t in throws),
config-v2                           | pants.engine.internals.scheduler.ExecutionError: 1 Exception encountered:
config-v2                           |
config-v2                           |   Exception: Failed to begin read transaction: Resource temporarily unavailable
config-v2                           |
config-v2                           |
config-v2                           | 16:24:25.23 [ERROR] Exception caught: (pants.engine.internals.scheduler.ExecutionError)
config-v2                           |   File "/home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.2_py36/lib/python3.6/site-packages/pants/bin/local_pants_runner.py", line 289, in run
config-v2                           |     engine_result = self._run_v2()
config-v2                           |   File "/home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.2_py36/lib/python3.6/site-packages/pants/bin/local_pants_runner.py", line 195, in _run_v2
config-v2                           |     return self._maybe_run_v2_body(goals, poll=False)
config-v2                           |   File "/home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.2_py36/lib/python3.6/site-packages/pants/bin/local_pants_runner.py", line 217, in _maybe_run_v2_body
config-v2                           |     poll_delay=(0.1 if poll else None),
config-v2                           |   File "/home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.2_py36/lib/python3.6/site-packages/pants/init/engine_initializer.py", line 127, in run_goal_rules
config-v2                           |     goal_product, params, poll=poll, poll_delay=poll_delay
config-v2                           |   File "/home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.2_py36/lib/python3.6/site-packages/pants/engine/internals/scheduler.py", line 569, in run_goal_rule
config-v2                           |     self._raise_on_error([t for _, t in throws])
config-v2                           |   File "/home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.2_py36/lib/python3.6/site-packages/pants/engine/internals/scheduler.py", line 539, in _raise_on_error
config-v2                           |     wrapped_exceptions=tuple(t.exc for t in throws),
config-v2                           |
config-v2                           | Exception message: 1 Exception encountered:
config-v2                           |
config-v2                           |   Exception: Failed to begin read transaction: Resource temporarily unavailable
config-v2                           |
config-v2                           |
config-v2                           | (Use --print-stacktrace to see more error details.)
config-v2                           | PID 61 waiting for a file lock (~/.pids/.lock.pantsd) held by: 60 (/home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.2_py36/bin/python /home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.2_py36/bin/pants --pants-bin-name=./pants --pants-version=2.0.2 run servers/api_runtime/src:runtime_server)
config-v2 exited with code 1
It almost looks like each container is trying to spin up its own instance of the daemon… ideally i want each container to share the daemon on the host… and then share the cache via binded mounts. theres got to be some setting or something im missing here.
heres my toml file …. maybe something will jump out.
Copy code
[GLOBAL]
level = "info"
pants_version = "2.0.2"
colors = true
pantsd = true
concurrent = false  # Run pants invocations and processes in parallel
process_execution_local_parallelism = 4

# Process Directories
pants_subprocessdir = "~/.pids"

[source]
root_patterns = [
    "/",
    "src",
]

[python-setup]
resolver_jobs = 2
interpreter_constraints = ['CPython>=3.6,<3.7']
requirement_constraints = '3rdparty/python/constraints.txt'
interpreter_search_paths = [
    '<PYENV>',
    '<PATH>',
]

[python-repos]
repos = [
    '%(buildroot)s/3rdparty/python/repo'
]
h

hundreds-father-404

06/25/2021, 5:10 PM
I see
pantsd = true
and
concurrent = false
. Are you disabling Pantsd another way?
s

salmon-barista-63163

06/25/2021, 5:16 PM
ah sorry i was messing with the settings trying it multiple ways.
Let me ask this: With the docker architecture we have where we run a single service with
pants run
in a container… how should i set these options? What is your opinion here for best performance
w

witty-crayon-22786

06/25/2021, 7:24 PM
tied up for a few hours, but then will respond here: sorry Brian
so, workflows involving running lots of copies of pants are just not advisable until we’ve resolved https://github.com/pantsbuild/pants/issues/7654 … i just got back from paternity leave, and will be able to resume it in the coming weeks (once the other blocker is resolved).
but even then, if pants is running in a bunch of containers, they’ll never be able to share resources across instances the way that
pantsd
depends on for performance
i think we’ve previously discussed using
./pants package
followed by running the package rather than
./pants run
, and i think that i would continue to recommend that for the “i need to run X binaries concurrently” case. you’d disable
pantsd
inside the container as well.
if they weren’t in separate docker containers, then https://github.com/pantsbuild/pants/issues/7654 would be helpful, because it would allow all of the resources to be shared across processes… but inside the container, there is no sharing.
4 Views