https://pantsbuild.org/ logo
Join Slack
Powered by
So, after upgrading to 2.33 (from 2.24, :cry: ) we...
# general
a
So, after upgrading to 2.33 (from 2.24, 😢 ) we've been unable to, sometimes, build packages on mac via docker. It's weird in that some binaries we can build, but not others. It turns out, it might be related to forums.docker.com/t/unix-socket-on-bind-mount/142653 - though, not really, since the socket is created in the container This is related to a pex upgrade (I don't know which version exactly introduced this, but confirmed that 2.97.1 works, can manually bisect if it really matters) that causes it to run multiprocessing sometimes to install packages and that uses a UNIX socket and that doesn't work. I guess the solution would be to see if it's possible for pex to not do that (using --jobs=1 doesn't work, still spawns a multiprocessing pool), but I was wondering if anyone has any better ideas.
Oh, right, this is the error:
Copy code
pex:       Installing 273 distributions: 240.3ms
pex:         Using 5 parallel jobs to install 22 wheels: 9.7ms
Traceback (most recent call last):
  File "/pants-named-caches/pex_root/installed_wheels/2/015e3985f214e986656c608f730e3d286e670df3fa711cfe5ac3580de021b320/pex-2.97.1-py2.py3-none-any.whl/pex/result.py", line 105, in catch
    return func(*args, **kwargs)
  File "/pants-named-caches/pex_root/installed_wheels/2/015e3985f214e986656c608f730e3d286e670df3fa711cfe5ac3580de021b320/pex-2.97.1-py2.py3-none-any.whl/pex/bin/pex.py", line 1435, in do_main
    pex_builder = build_pex(
        requirement_configuration=requirement_configuration,
    ...<3 lines>...
        options=options,
    )
  File "/pants-named-caches/pex_root/installed_wheels/2/015e3985f214e986656c608f730e3d286e670df3fa711cfe5ac3580de021b320/pex-2.97.1-py2.py3-none-any.whl/pex/bin/pex.py", line 1111, in build_pex
    resolve_result = resolve(
        targets=targets,
    ...<9 lines>...
        dependency_configuration=dependency_config,
    )
  File "/pants-named-caches/pex_root/installed_wheels/2/015e3985f214e986656c608f730e3d286e670df3fa711cfe5ac3580de021b320/pex-2.97.1-py2.py3-none-any.whl/pex/resolve/configured_resolve.py", line 172, in resolve
    return resolve_via_pip(
        targets=targets,
    ...<20 lines>...
        dependency_configuration=dependency_configuration,
    )
  File "/pants-named-caches/pex_root/installed_wheels/2/015e3985f214e986656c608f730e3d286e670df3fa711cfe5ac3580de021b320/pex-2.97.1-py2.py3-none-any.whl/pex/resolver.py", line 1710, in resolve
    build_and_install_request.install_distributions(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        ignore_errors=ignore_errors, max_parallel_jobs=max_parallel_jobs
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/pants-named-caches/pex_root/installed_wheels/2/015e3985f214e986656c608f730e3d286e670df3fa711cfe5ac3580de021b320/pex-2.97.1-py2.py3-none-any.whl/pex/resolver.py", line 1513, in install_distributions
    for install_result in iter_map_parallel(
                          ~~~~~~~~~~~~~~~~~^
        inputs=install_requests,
        ^^^^^^^^^^^^^^^^^^^^^^^^
    ...<6 lines>...
        result_render_function=InstallResult.wheel_file,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ):
    ^
  File "/pants-named-caches/pex_root/installed_wheels/2/015e3985f214e986656c608f730e3d286e670df3fa711cfe5ac3580de021b320/pex-2.97.1-py2.py3-none-any.whl/pex/jobs.py", line 788, in iter_map_parallel
    with _mp_pool(size=pool_size) as pool:
         ~~~~~~~~^^^^^^^^^^^^^^^^
  File "/pants-named-caches/python_build_standalone/39fa30435b5f9f2e3ed08a1247c0a6527fce9d8511437a3ed71b1dff568b92e6/lib/python3.14/contextlib.py", line 141, in __enter__
    return next(self.gen)
  File "/pants-named-caches/pex_root/installed_wheels/2/015e3985f214e986656c608f730e3d286e670df3fa711cfe5ac3580de021b320/pex-2.97.1-py2.py3-none-any.whl/pex/jobs.py", line 704, in _mp_pool
    pool = multiprocessing.Pool(processes=size)
  File "/pants-named-caches/python_build_standalone/39fa30435b5f9f2e3ed08a1247c0a6527fce9d8511437a3ed71b1dff568b92e6/lib/python3.14/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
                context=self.get_context())
  File "/pants-named-caches/python_build_standalone/39fa30435b5f9f2e3ed08a1247c0a6527fce9d8511437a3ed71b1dff568b92e6/lib/python3.14/multiprocessing/pool.py", line 215, in __init__
    self._repopulate_pool()
    ~~~~~~~~~~~~~~~~~~~~~^^
  File "/pants-named-caches/python_build_standalone/39fa30435b5f9f2e3ed08a1247c0a6527fce9d8511437a3ed71b1dff568b92e6/lib/python3.14/multiprocessing/pool.py", line 306, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
                                        self._processes,
                                        ^^^^^^^^^^^^^^^^
    ...<3 lines>...
                                        self._maxtasksperchild,
                                        ^^^^^^^^^^^^^^^^^^^^^^^
                                        self._wrap_exception)
                                        ^^^^^^^^^^^^^^^^^^^^^
  File "/pants-named-caches/python_build_standalone/39fa30435b5f9f2e3ed08a1247c0a6527fce9d8511437a3ed71b1dff568b92e6/lib/python3.14/multiprocessing/pool.py", line 329, in _repopulate_pool_static
    w.start()
    ~~~~~~~^^
  File "/pants-named-caches/python_build_standalone/39fa30435b5f9f2e3ed08a1247c0a6527fce9d8511437a3ed71b1dff568b92e6/lib/python3.14/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ~~~~~~~~~~~^^^^^^
  File "/pants-named-caches/python_build_standalone/39fa30435b5f9f2e3ed08a1247c0a6527fce9d8511437a3ed71b1dff568b92e6/lib/python3.14/multiprocessing/context.py", line 300, in _Popen
    return Popen(process_obj)
  File "/pants-named-caches/python_build_standalone/39fa30435b5f9f2e3ed08a1247c0a6527fce9d8511437a3ed71b1dff568b92e6/lib/python3.14/multiprocessing/popen_forkserver.py", line 35, in __init__
    super().__init__(process_obj)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/pants-named-caches/python_build_standalone/39fa30435b5f9f2e3ed08a1247c0a6527fce9d8511437a3ed71b1dff568b92e6/lib/python3.14/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
    ~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/pants-named-caches/python_build_standalone/39fa30435b5f9f2e3ed08a1247c0a6527fce9d8511437a3ed71b1dff568b92e6/lib/python3.14/multiprocessing/popen_forkserver.py", line 51, in _launch
    self.sentinel, w = forkserver.connect_to_new_process(self._fds)
                       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/pants-named-caches/python_build_standalone/39fa30435b5f9f2e3ed08a1247c0a6527fce9d8511437a3ed71b1dff568b92e6/lib/python3.14/multiprocessing/forkserver.py", line 89, in connect_to_new_process
    self.ensure_running()
    ~~~~~~~~~~~~~~~~~~~^^
  File "/pants-named-caches/python_build_standalone/39fa30435b5f9f2e3ed08a1247c0a6527fce9d8511437a3ed71b1dff568b92e6/lib/python3.14/multiprocessing/forkserver.py", line 159, in ensure_running
    os.chmod(address, 0o600)
    ~~~~~~~~^^^^^^^^^^^^^^^^
OSError: [Errno 22] Invalid argument: '/pants-sandbox/pants-sandbox-uOpBP2/.tmp/pymp-j_z_u9v6/sock-d3zx1djz'
[Errno 22] Invalid argument: '/pants-sandbox/pants-sandbox-uOpBP2/.tmp/pymp-j_z_u9v6/sock-d3zx1djz'
And 2.97.1:
Copy code
pex:       Installing 273 distributions: 984.2ms
pex:         Using 5 parallel jobs to install 22 wheels: 530.9ms
pex:       Checking install: 85.5ms
pex:   Configuring PEX dependencies: 151.0ms
Saving PEX file to creek/bin-creek.pex
Wait.
I... copied that from the wrong docs.
I wonder if it's a 3.14 change, not pex
Yeah, that seems to be it. 😢
Okay, I don't know what it is, because now I cannot force it to use parallel jobs
I think I might open an issue against pex tomorrow, the fact that this is not controllable or even deterministic (as far as I can tell) is kind of annoying.
b
@ancient-france-42909 if you do report to Pex, please do your homework first and make sure this is a Pex issue and not a Pants issue. My code read quickly led me here: 1. github.com/pantsbuild/pants/blob/…/docker.rs#… 2. github.com/pantsbuild/pants/blob/…/docker.rs#… So 
/pants-named-caches/
is a volume mount from the host machine. With that, page in these: + github.com/docker/for-mac/issues/6614 + docs.python.org/3/whatsnew/…#…
On Unix platforms other than macOS, ‘forkserver’ is now the default start method (replacing ‘fork’). This change does not affect Windows or macOS, where ‘spawn’ remains the default start method.
So ... afaict there is no Pex bug. There is certainly a Docker bug on macOS (linked above). And exactly that bug would be triggered by a switch from mp fork to mp forkserver, which is a new 3.14 thing for Linux. I will point out I knew 0 here and just used your backtrace and some googling.
It turns out, it might be related to forums.docker.com/t/unix-socket-on-bind-mount/142653 - though, not really, since the socket is created in the container
"though, not really, since the socket is created in the container" <- by "the socket", what are you referring to? If 
/pants-sandbox/pants-sandbox-uOpBP2/.tmp/pymp-j_z_u9v6/sock-d3zx1djz
, that socket used by the mp forkserver for communications is not in the container - its on the 
/pants-named-caches
bind mount.
a
I misread that docker community forums thing. But, I didn't mean this was a bug in pex, just that the inconsistency in when multiprocessing is used is a bit frustrating.
b
So you agree the proximal bug is old and is in Docker for Mac. You upgraded at least 2 things, Pants and Python, and its the Python bit that is the next most proximal cause and you're focusing on when mp is or is not used as the frustrating thing?
a
Technically, 3, but I'm fairly sure it's not pex that's the problem. But, if I could disable that in pex, that'd be easier than getting docker to fix the bug or changing how pants does things. And, as I said, I was going to continue looking into this tomorrow, maybe I should've phrased it differently, but it was 10:30pm and I went to do something else for the night.
b
If you want to propose a feature to Pex to allow user-selection of mp method, that's fine - I'm flexible and can help you work around those bugs.
a
I didn't mean to say the fault lied with pex, sorry
b
I will say, the fact chmod doesn;t work in a Pants docker sandbox for mac - is a much bigger and wider problem.
a
It's not chmod that doesn't work, anything on unix sockets
b
Its fairly stunning if this hasn't been hit before in Python or any other process lang running in a sandbox.
a
That thread talks about ls failing, heh
"you created a file and want to stat it? naaah"
b
Right. That is narrower.
At any rate, I'll just go ahead and add this knob and it should be released before you wake up. Please though, maybe report this to Pants. Even though unix socket + chmod is narrow, it suggests docker runner for mac has a pretty serious hole that should at least be FAQ'd.
a
True.
That's why I originally asked if anyone knows of something I can do without changes in other software
b
And if you don't actually write mac software at your company, please file a bug there against its use of macs.
Its braindamaged. True, mass-braindamage, but really dumb nonetheless.
a
I agree. A coworker said:
can i get my thinkpad back
b
Asahi works well, I used for a bit, but that's stuck on M2's and older.
a
Yeah, no, it's the OS they want, for MDM
b
What is an MDM?
a
Ah, device management. I think it's Mobile Device Management or something.
b
Jesus, yeah. Makes "sense".
a
iOS, iPadOS, macOS and tvOS have a built-in framework that supports mobile device management (MDM). MDM lets you securely and wirelessly configure devices by sending profiles and commands to the device, whether they’re owned by the user or your organisation. MDM capabilities include updating software and device settings, monitoring compliance with organisational policies, and remotely wiping or locking devices
And they just don't want to bother with Linux. Before we were acquired, we had something okay for that, but 🤷
b
github.com/pex-tool/pex/issues/3256 - I'll fill out more details and use that to track the PR / release.
a
Thank you very much, I'll make one for pants and comment there with the reasons, but it really is late here, sorry 😞 I need to be up in 6 hours so I'm going to go sleep.
b
https://github.com/pex-tool/pex/releases/tag/v2.101.0 You should be able to upgrade Pex and then plumb the PEX_MULTIPROCESSING_START_METHOD env var through to the docker executor (not sure how you do that, but Pants folks can probably help). You probably want a value of 
spawn
to be safe. That said, I also fixed 
--jobs
and 
--max-install-jobs
to skip multiprocessing if set to 
1
.
a
Sorry, it was a very long day at work today, I couldn't check this until now. That's amazing
Pants does allow you to pass cmdline args to pex
But it's again very late, so I probably won't be able to try it tonight, I will try to thought.
Thank you!
b
Pants does allow you to pass cmdline args to pex
This is only available via env var, not a command line arg (unless you mean restricting jobs to 1 - but that's a crappy option IMO). Regardless, it does look like Pants allows passing the env var through one of the many env var options listed for pantsbuild.org/2.32/reference/targets/docker_environment
a
We're going to only apply that for people's local envs. And, honestly, resolving deps takes almost 2 minutes, installing the wheels takes 1-2 seconds, we're fine
Hey, thank you, again. Setting the multiprocessing one didn't seem to work, but we just set jobs to 1 and that works. Later tonight I'll make a pants ticket
b
You're welcome. > Setting the multiprocessing one didn't seem to work > That's a bit vague. Here I prove the setting does work in a Pants docker environment context - 
spawn
is used as the start method in this example: github.com/jsirois/pants-PEX_MU… It's true getting a repro is tricky, you must nuke pants named cache volumes and ensure a fresh container is started to observe 
PEX_MULTIPROCESSING_START_METHOD=spawn
is plumbed to the container and seen by Pex there.
a
A coworker tried to use spawn and it still crashed. jobs=1 works, so whatever. However, heh... We cannot use this, yet. 😞 I have to go and change all our apps to not do stupid stuff like extract the venv as root, or add the env var to not try to write 
.last_access
when starting.
b
A coworker tried to use spawn and it still crashed. jobs=1 works, so whatever.
FWIW I put actual effort into the fixes / release; so I do not take this as "so whatever". When you throw stuff over the wall and actually get a response back - its nice to meet the effort with at least equal effort since this is free labor.
Open in Slack
PreviousNext
Powered by