Hello So I am in the process of moving our main repo over to Pants #general

Hello. So I am in the process of moving our main r...

salmon-barista-63163

06/24/2021, 10:18 PM

Hello. So I am in the process of moving our main repo over to pants 2. Our “dev” environment that is run for each developer runs as follows. • centOS ec2 instance that runs a set of docker containers • each docker container uses the same base image • Pants (along with all of thee code) is bind mounted into the containers • the entrypoint and command for each docker container is a

./pants run

command. This all worked great with pants v1 and we mounted in the cache so everything was quick and worked well. With pants 2 if i start up one container everything works just fine, when we start up more than 1 container i get the following error.

Copy code

config-v1                           | 22:06:54.17 [ERROR] Exception caught: (pants.engine.internals.scheduler.ExecutionError)
config-v1                           |   File "/home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.2_py36/lib/python3.6/site-packages/pants/bin/local_pants_runner.py", line 289, in run
config-v1                           |     engine_result = self._run_v2()
config-v1                           |   File "/home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.2_py36/lib/python3.6/site-packages/pants/bin/local_pants_runner.py", line 195, in _run_v2
config-v1                           |     return self._maybe_run_v2_body(goals, poll=False)
config-v1                           |   File "/home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.2_py36/lib/python3.6/site-packages/pants/bin/local_pants_runner.py", line 217, in _maybe_run_v2_body
config-v1                           |     poll_delay=(0.1 if poll else None),
config-v1                           |   File "/home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.2_py36/lib/python3.6/site-packages/pants/init/engine_initializer.py", line 127, in run_goal_rules
config-v1                           |     goal_product, params, poll=poll, poll_delay=poll_delay
config-v1                           |   File "/home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.2_py36/lib/python3.6/site-packages/pants/engine/internals/scheduler.py", line 569, in run_goal_rule
config-v1                           |     self._raise_on_error([t for _, t in throws])
config-v1                           |   File "/home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.2_py36/lib/python3.6/site-packages/pants/engine/internals/scheduler.py", line 539, in _raise_on_error
config-v1                           |     wrapped_exceptions=tuple(t.exc for t in throws),
config-v1                           |
config-v1                           | Exception message: 1 Exception encountered:
config-v1                           |
config-v1                           |   Exception: Failed to begin read transaction: Resource temporarily unavailable

I suspect this has something to do with how we are mounting in the cache dirs: IE we mount

~/.cache/pants -> ~/.cache/pants

If we do not mount in the cache all the services come up no problem but each has to build and resolve all the constraints first which takes a long time. Any thoughts here? Am I missing something? Any suggestions on what to try to resolve this?

happy-kitchen-89482

06/25/2021, 1:03 AM

Hmmm, what happens if you run without pantsd (

--no-pantsd

) ?

salmon-barista-63163

06/25/2021, 12:04 PM

This is without pantsd @happy-kitchen-89482

happy-kitchen-89482

06/25/2021, 4:26 PM

Huh. ccing @witty-crayon-22786 for later (after his build meetup talk)

salmon-barista-63163

06/25/2021, 4:30 PM

Yeah if i have pantsd FALSE and concurrent TRUE this starts to happen. with pantsd TRUE and concurrent FALSE I get errors like this:

Copy code

config-v2                           | 16:24:24.10 [INFO] initializing pantsd...
mock-stitch-api exited with code 1
config-v2                           | 16:24:25.08 [INFO] pantsd initialized.
config-v2                           | Exception in thread SchedulerServiceThread:
config-v2                           | Traceback (most recent call last):
config-v2                           |   File "/home/centos/.pyenv/versions/3.6.12/lib/python3.6/threading.py", line 916, in _bootstrap_inner
config-v2                           |     self.run()
config-v2                           |   File "/home/centos/.pyenv/versions/3.6.12/lib/python3.6/threading.py", line 864, in run
config-v2                           |     self._target(*self._args, **self._kwargs)
config-v2                           |   File "/home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.2_py36/lib/python3.6/site-packages/pants/pantsd/service/pants_service.py", line 219, in target
config-v2                           |     service.run()
config-v2                           |   File "/home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.2_py36/lib/python3.6/site-packages/pants/pantsd/service/scheduler_service.py", line 141, in run
config-v2                           |     self._invalidation_globs_and_snapshot = (globs, self._get_snapshot(globs, poll=False))
config-v2                           |   File "/home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.2_py36/lib/python3.6/site-packages/pants/pantsd/service/scheduler_service.py", line 85, in _get_snapshot
config-v2                           |     timeout=timeout,
config-v2                           |   File "/home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.2_py36/lib/python3.6/site-packages/pants/engine/internals/scheduler.py", line 593, in product_request
config-v2                           |     self._raise_on_error([t for _, t in throws])
config-v2                           |   File "/home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.2_py36/lib/python3.6/site-packages/pants/engine/internals/scheduler.py", line 539, in _raise_on_error
config-v2                           |     wrapped_exceptions=tuple(t.exc for t in throws),
config-v2                           | pants.engine.internals.scheduler.ExecutionError: 1 Exception encountered:
config-v2                           |
config-v2                           |   Exception: Failed to begin read transaction: Resource temporarily unavailable
config-v2                           |
config-v2                           |
config-v2                           | 16:24:25.23 [ERROR] Exception caught: (pants.engine.internals.scheduler.ExecutionError)
config-v2                           |   File "/home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.2_py36/lib/python3.6/site-packages/pants/bin/local_pants_runner.py", line 289, in run
config-v2                           |     engine_result = self._run_v2()
config-v2                           |   File "/home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.2_py36/lib/python3.6/site-packages/pants/bin/local_pants_runner.py", line 195, in _run_v2
config-v2                           |     return self._maybe_run_v2_body(goals, poll=False)
config-v2                           |   File "/home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.2_py36/lib/python3.6/site-packages/pants/bin/local_pants_runner.py", line 217, in _maybe_run_v2_body
config-v2                           |     poll_delay=(0.1 if poll else None),
config-v2                           |   File "/home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.2_py36/lib/python3.6/site-packages/pants/init/engine_initializer.py", line 127, in run_goal_rules
config-v2                           |     goal_product, params, poll=poll, poll_delay=poll_delay
config-v2                           |   File "/home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.2_py36/lib/python3.6/site-packages/pants/engine/internals/scheduler.py", line 569, in run_goal_rule
config-v2                           |     self._raise_on_error([t for _, t in throws])
config-v2                           |   File "/home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.2_py36/lib/python3.6/site-packages/pants/engine/internals/scheduler.py", line 539, in _raise_on_error
config-v2                           |     wrapped_exceptions=tuple(t.exc for t in throws),
config-v2                           |
config-v2                           | Exception message: 1 Exception encountered:
config-v2                           |
config-v2                           |   Exception: Failed to begin read transaction: Resource temporarily unavailable
config-v2                           |
config-v2                           |
config-v2                           | (Use --print-stacktrace to see more error details.)
config-v2                           | PID 61 waiting for a file lock (~/.pids/.lock.pantsd) held by: 60 (/home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.2_py36/bin/python /home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.2_py36/bin/pants --pants-bin-name=./pants --pants-version=2.0.2 run servers/api_runtime/src:runtime_server)
config-v2 exited with code 1

salmon-barista-63163

06/25/2021, 4:36 PM

It almost looks like each container is trying to spin up its own instance of the daemon… ideally i want each container to share the daemon on the host… and then share the cache via binded mounts. theres got to be some setting or something im missing here.

salmon-barista-63163

06/25/2021, 5:01 PM

heres my toml file …. maybe something will jump out.

Copy code

[GLOBAL]
level = "info"
pants_version = "2.0.2"
colors = true
pantsd = true
concurrent = false  # Run pants invocations and processes in parallel
process_execution_local_parallelism = 4

# Process Directories
pants_subprocessdir = "~/.pids"

[source]
root_patterns = [
    "/",
    "src",
]

[python-setup]
resolver_jobs = 2
interpreter_constraints = ['CPython>=3.6,<3.7']
requirement_constraints = '3rdparty/python/constraints.txt'
interpreter_search_paths = [
    '<PYENV>',
    '<PATH>',
]

[python-repos]
repos = [
    '%(buildroot)s/3rdparty/python/repo'
]

hundreds-father-404

06/25/2021, 5:10 PM

I see

pantsd = true

and

concurrent = false

. Are you disabling Pantsd another way?

salmon-barista-63163

06/25/2021, 5:16 PM

ah sorry i was messing with the settings trying it multiple ways.

salmon-barista-63163

06/25/2021, 5:17 PM

Let me ask this: With the docker architecture we have where we run a single service with

pants run

in a container… how should i set these options? What is your opinion here for best performance

witty-crayon-22786

06/25/2021, 7:24 PM

tied up for a few hours, but then will respond here: sorry Brian

witty-crayon-22786

06/25/2021, 8:31 PM

so, workflows involving running lots of copies of pants are just not advisable until we’ve resolved https://github.com/pantsbuild/pants/issues/7654 … i just got back from paternity leave, and will be able to resume it in the coming weeks (once the other blocker is resolved).

witty-crayon-22786

06/25/2021, 8:32 PM

but even then, if pants is running in a bunch of containers, they’ll never be able to share resources across instances the way that

pantsd

depends on for performance

witty-crayon-22786

06/25/2021, 8:33 PM

i think we’ve previously discussed using

./pants package

followed by running the package rather than

./pants run

, and i think that i would continue to recommend that for the “i need to run X binaries concurrently” case. you’d disable

pantsd

inside the container as well.

witty-crayon-22786

06/25/2021, 8:34 PM

if they weren’t in separate docker containers, then https://github.com/pantsbuild/pants/issues/7654 would be helpful, because it would allow all of the resources to be shared across processes… but inside the container, there is no sharing.

4 Views

Open in Slack

Previous Next