@brief-scientist-13682 👋 hello I don't think aiden is in here; but if you want to talk about github.com/pex-tool/pex/pull/3262 and what we're seeing I'm around

If you are using the uv resolver, then this is likely github.com/pantsbuild/pants/issues/23657 I'd like to verify or deny that 1st.

Nope EBS and instance-store only

Ok, well I'll need more details likely, but the fix in the PR is patchy - just addressed

--layout packed

and not loose or zipapp, and getting to the bottom of the corruption is the preferred route.

Yeah for sure

So Pex will fail in environments where flock doesn't work (NFS angle), and it will fail if fed bad input wheels, or if it has external party delete from its cache. But those are the only known cases over many years.

Yeah that makes absolute sense. AFAICT neither of those is the case that we're seeing here.

IIUC your company is ~long time Pants user; so what changed?

when we upgraded we moved to pex 2.97.3

That's fishy. Why and why that version in particular.

Latest patch of the version that pants wanted to be on that was all.

Ok, so just lucky timing.

Yeah we upgraded on the 20th IIRC

Ok, well, then there is no known bug that would cause Pex to write bad atomic directories; so the existing PR just patches over the unknown cause and only does it for packed. Those are both problems. I'm trying to think what I might ask you to provide that could help me debug.

Yes we do. I have reason to believe what brought this to our attention was that I fixed our cache's deployment to be much more stable (ie not crashlooping). This then meant that builds were actually able to read/write consistently so when the bad pex got built and pushed to the cache it was then distributed to other builds.

Ok. So that I know the full code path here, these are useful to answer: + What

pip_version

do you set in pants.toml if any? + What

interpreter_constraints

are in-play? + Do you enable resolves (locks)? Basically, the more of pants.toml I can see the better.

Roger that. 👀

Ok, and I typo-ed and corrected above - sorry - its pip_version I need - you already let me know Pex was 2.97.3

Ah 👀

And no ICs?

pants.toml [python]

[python]
enable_resolves = true
resolves_generate_lockfiles = true

interpreter_constraints = [">=3.10,<3.13"]
tailor_requirements_targets = false

In CI, do you know which is the real Python version used? If you do not set

pip_version

the version used depends on the Python version selected by Pex from the range.

Ok yeah we don't set it. It should be py 3.10 😭

Ok, I'll assume 3.10 which means Pex picks Pip 26.1.

yes

Pants sandbox right?

The linked option above.

Ok no we don't. Thought i have seen

pants-sandbox

dirs around. Which is why i initially said yes

Yeah - there is a name overload there. Ok, great - thanks for the info. I'll spend a few hours staring at the code to see how there could be a write bug and report back.

One thing I was looking at is Pant's request hedging and cancellation of pex procs

That has existed for a very long time and should not be new. What was your Pants version upgrade from?

Pants went from 2.31.0 -> 2.33.0. We wanted some of the perf changes as well as the opentelemetry trace support.

Last night we finally were able to find a build that produced such a malformed artifact and persisted it into the remote-cache rather than consuming one.

Good datapoint. How do you know the build produced the artifact? This was a build that had 0 local pants caches?

Correct it cache missed on the build step for this - I will try to get you as many specifics as I'm able

A remote cache miss can pull from local cache; i.e. ~/.cache/pants/named-caches which Pex uses

Yeah we've disabled local caches entirely at this point and have just remote going.

Yeah we've disabled local caches entirely

What is the config that does that?

Ok cool - yeah we've disabled pants' local cache opt. I do not think we've disabled pex's

Ok. So Pex may or may not be pulling from local cache. That would have to be a volume mount into the container though.

I have just discovered as we're speaking here that it is in fact shared from the host 😭

Ok, and what type of sharing is this linux host, linux container, windows host linux container - and what is the the volume mount technology being used?

Linux bind mount -> docker

Sorry - fs driver for docker

no worries we're both learning some things about the here 🙃

I think the docker fs driver story has settled, but in the early days there were ones you could pick IIRC even on linux that would not respect locks

It def won't have pre-pex2.97 artifacts because we've fully rolled the fleet of nodes

Ok.

But there might be other shenanigans like a build getting oomkilled or something weird

Well, Pex should be resilient to that.

I was able to repro this failure by having ~30 pex procs in parallel building the same pex and killing ~1/3 of them. But I don't trust that this is representative of what actually happens to us

Please provide the script!

well when I say I - i mean a robot that I haven't validated the work of; didn't want to pass you something I didn't personally review.

I hate AI. Ok, but what you're relaying is what it claimed?

Ok, but what you're relaying is what it claimed?

Yes, it had something that it managed to repro with. Once I get you the docker fs question answer I'll actually review what it has.

Ok, great. I'll work on reproing the repro.

We're using overlay2. The builds are running on

m8gd.4xlarge

and

m8id.4xlarge

hosts in AWS. All the disk at play here should be disk from the dedicated NVME side of those nodes (ofc they still have a root EBS as well)

1st reading this: docs.docker.com/engine/storage/drivers/overlayfs-driver#…

More changes to the story here, sorry. It appears all of our apps that have run into this are not using our repo's configured python 3.10 but are instead on 3.12 Which changes the pip pex end up using right?

I think that will also select 26.1. Thanks for the info though. I will note you should be selecting the latest Pip supported though instead of relying on defaulting. Pip has been getting love in the last 1.5 years and is getting faster.

Roger that - given my core mission of build speed that's super good intel 🙂

So that I'm clear - still reading up on overlayfs2 - a single host named caches is shared to 1 container or multiple concurrent containers?

1 container at a time.

Yes. Yeah Pants / Pex parallelism on a normal host has been hammered for a long time. I'm just probing the point of the flock'ing. If the flock is questionable all bets are off.

OK I gotta go do a few weekend chores and such. But i'll be back in a few hours / be on mobile.

I would love that AI claimed repro. I can't repro it yet. Using:

#!/usr/bin/env bash

set -euo pipefail

COUNT=30
MOD=3

export PEX_ROOT="${PEX_ROOT:-/tmp/pex-root}"

pex3 lock create --pip-version latest-compatible ansible --indent 2 -o /tmp/lock.json
rm -rf "${PEX_ROOT}"

declare -a kill_pex_pids
declare -a wait_pex_pids
for ((x=1; x<COUNT; x++)); do
    pex ansible -c ansible --lock /tmp/lock.json --venv prepend --layout packed -- --version &
    pid=$!
    if (( x % MOD == 0 )); then
        kill_pex_pids+=($pid)
    else
        wait_pex_pids+=($pid)
    fi
done

for pid in "${kill_pex_pids[@]}"; do
    sleep 2
    kill ${pid}
done

for pid in "${wait_pex_pids[@]}"; do
    wait ${pid}
done

I get things like this with exit code 0:

/home/jsirois/bin/tools.venv/lib/python3.14/site-packages/pex/atomic_directory.py:268: PEXWarning: [pid:60339, tid:131625590040256, cwd:/home/jsirois/support/pex/slack/8-29-2026]: After obtaining an exclusive lock on /tmp/pex-root/downloads/2/.fb06b66c8da04172d9e72a21d7d06186d8919e32ae5ab5cdf5b9d920be805ac2.atomic_directory.lck, failed to establish a work directory at /tmp/pex-root/downloads/2/fb06b66c8da04172d9e72a21d7d06186d8919e32ae5ab5cdf5b9d920be805ac2.lck.work due to: [Errno 17] File exists: '/tmp/pex-root/downloads/2/fb06b66c8da04172d9e72a21d7d06186d8919e32ae5ab5cdf5b9d920be805ac2.lck.work'
  pex_warnings.warn(
/home/jsirois/bin/tools.venv/lib/python3.14/site-packages/pex/atomic_directory.py:268: PEXWarning: [pid:60347, tid:126699643143872, cwd:/home/jsirois/support/pex/slack/8-29-2026]: After obtaining an exclusive lock on /tmp/pex-root/downloads/2/.9e7dd367f7dc5d5e9fc5ae1baf8af9c4edc09e916a73a40108a3f32e3ad93f10.atomic_directory.lck, failed to establish a work directory at /tmp/pex-root/downloads/2/9e7dd367f7dc5d5e9fc5ae1baf8af9c4edc09e916a73a40108a3f32e3ad93f10.lck.work due to: [Errno 17] File exists: '/tmp/pex-root/downloads/2/9e7dd367f7dc5d5e9fc5ae1baf8af9c4edc09e916a73a40108a3f32e3ad93f10.lck.work'
  pex_warnings.warn(
/home/jsirois/bin/tools.venv/lib/python3.14/site-packages/pex/atomic_directory.py:279: PEXWarning: [pid:60339, tid:131625590040256, cwd:/home/jsirois/support/pex/slack/8-29-2026]: Continuing to forcibly re-create the work directory at /tmp/pex-root/downloads/2/fb06b66c8da04172d9e72a21d7d06186d8919e32ae5ab5cdf5b9d920be805ac2.lck.work.
  pex_warnings.warn(
/home/jsirois/bin/tools.venv/lib/python3.14/site-packages/pex/atomic_directory.py:279: PEXWarning: [pid:60347, tid:126699643143872, cwd:/home/jsirois/support/pex/slack/8-29-2026]: Continuing to forcibly re-create the work directory at /tmp/pex-root/downloads/2/9e7dd367f7dc5d5e9fc5ae1baf8af9c4edc09e916a73a40108a3f32e3ad93f10.lck.work.
  pex_warnings.warn(
/home/jsirois/.pyenv/versions/3.14.7/lib/python3.14/multiprocessing/resource_tracker.py:475: UserWarning: resource_tracker: There appear to be 2 leaked semaphore objects to clean up at shutdown: {'/mp-eleeewhf', '/mp-inn89lbx'}
  warnings.warn(
/home/jsirois/.pyenv/versions/3.14.7/lib/python3.14/multiprocessing/resource_tracker.py:475: UserWarning: resource_tracker: There appear to be 2 leaked semaphore objects to clean up at shutdown: {'/mp-r2k9mixn', '/mp-fv5ir0r1'}
  warnings.warn(
/home/jsirois/.pyenv/versions/3.14.7/lib/python3.14/multiprocessing/resource_tracker.py:475: UserWarning: resource_tracker: There appear to be 2 leaked semaphore objects to clean up at shutdown: {'/mp-qg5m83q7', '/mp-wtniolvk'}
  warnings.warn(
/home/jsirois/.pyenv/versions/3.14.7/lib/python3.14/multiprocessing/resource_tracker.py:475: UserWarning: resource_tracker: There appear to be 2 leaked semaphore objects to clean up at shutdown: {'/mp-7oaiujg4', '/mp-yawzw6h2'}
  warnings.warn(
/home/jsirois/.pyenv/versions/3.14.7/lib/python3.14/multiprocessing/resource_tracker.py:475: UserWarning: resource_tracker: There appear to be 6 leaked semaphore objects to clean up at shutdown: {'/mp-3elo32t7', '/mp-jb8j0soh', '/mp-__qceg7m', '/mp-19byohln', '/mp-7xca8xo8', '/mp-u8yvl5h8'}
  warnings.warn(
Process ForkServerPoolWorker-1:
Traceback (most recent call last):
  File "/home/jsirois/.pyenv/versions/3.14.7/lib/python3.14/multiprocessing/pool.py", line 131, in worker
    put((job, i, result))
    ~~~^^^^^^^^^^^^^^^^^^
  File "/home/jsirois/.pyenv/versions/3.14.7/lib/python3.14/multiprocessing/queues.py", line 397, in put
    self._writer.send_bytes(obj)
    ~~~~~~~~~~~~~~~~~~~~~~~^^^^^
  File "/home/jsirois/.pyenv/versions/3.14.7/lib/python3.14/multiprocessing/connection.py", line 210, in send_bytes
    self._send_bytes(m[offset:offset + size])
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jsirois/.pyenv/versions/3.14.7/lib/python3.14/multiprocessing/connection.py", line 441, in _send_bytes
    self._send(header)
    ~~~~~~~~~~^^^^^^^^
  File "/home/jsirois/.pyenv/versions/3.14.7/lib/python3.14/multiprocessing/connection.py", line 404, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
...
ansible [core 2.21.3]
  config file = None
  configured module search path = ['/home/jsirois/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /tmp/pex-root/venvs/3/0006b71e2cfba8fe25d1d711b7e930f1aec28f6f/c40de391655dede9f50dced114f8a1dc8f21dd91/lib/python3.14/site-packages/ansible
  ansible collection location = /home/jsirois/.ansible/collections:/usr/share/ansible/collections
  executable location = /tmp/pex-root/venvs/3/0006b71e2cfba8fe25d1d711b7e930f1aec28f6f/c40de391655dede9f50dced114f8a1dc8f21dd91/pex
  python version = 3.14.7 (main, Aug  5 2026, 17:40:48) [GCC 15.2.0] (/tmp/pex-root/venvs/3/0006b71e2cfba8fe25d1d711b7e930f1aec28f6f/c40de391655dede9f50dced114f8a1dc8f21dd91/bin/python)
  jinja version = 3.1.6
  pyyaml version = 6.0.3 (with libyaml v0.2.5)ansible [core 2.21.3]
  config file = None
  configured module search path = ['/home/jsirois/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /tmp/pex-root/venvs/3/0006b71e2cfba8fe25d1d711b7e930f1aec28f6f/c40de391655dede9f50dced114f8a1dc8f21dd91/lib/python3.14/site-packages/ansible
  ansible collection location = /home/jsirois/.ansible/collections:/usr/share/ansible/collections
  executable location = /tmp/pex-root/venvs/3/0006b71e2cfba8fe25d1d711b7e930f1aec28f6f/c40de391655dede9f50dced114f8a1dc8f21dd91/pex
  python version = 3.14.7 (main, Aug  5 2026, 17:40:48) [GCC 15.2.0] (/tmp/pex-root/venvs/3/0006b71e2cfba8fe25d1d711b7e930f1aec28f6f/c40de391655dede9f50dced114f8a1dc8f21dd91/bin/python)
  jinja version = 3.1.6
  pyyaml version = 6.0.3 (with libyaml v0.2.5)
ansible [core 2.21.3]
  config file = None
  configured module search path = ['/home/jsirois/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /tmp/pex-root/venvs/3/0006b71e2cfba8fe25d1d711b7e930f1aec28f6f/c40de391655dede9f50dced114f8a1dc8f21dd91/lib/python3.14/site-packages/ansible
  ansible collection location = /home/jsirois/.ansible/collections:/usr/share/ansible/collections
  executable location = /tmp/pex-root/venvs/3/0006b71e2cfba8fe25d1d711b7e930f1aec28f6f/c40de391655dede9f50dced114f8a1dc8f21dd91/pex
  python version = 3.14.7 (main, Aug  5 2026, 17:40:48) [GCC 15.2.0] (/tmp/pex-root/venvs/3/0006b71e2cfba8fe25d1d711b7e930f1aec28f6f/c40de391655dede9f50dced114f8a1dc8f21dd91/bin/python)
  jinja version = 3.1.6
  pyyaml version = 6.0.3 (with libyaml v0.2.5)
...

N.B.: This has the

failed to establish a work directory at ... Continuing to forcibly re-create the work directory at

logging @echoing-psychiatrist-92346 reported in the PR due to the kills as well as multiprocessing complaining about kills, but all the non-killed Pex processes run to successful completion.

😞 Mac Docker Desktop uses FUSE to mount local APFS directories into containers - the repro was on that FUSE fs. Not in an ext4 bindmount, or the overlayfs2. Since this is not a repro of what we're seeing in production I will keep hammering on this and come back when I've got a useable repro for you.

Ok. Thank you. My docker wrapper repro did no repro, but I'm Linux / Linux:

#!/usr/bin/env bash

set -euo pipefail

CONTAINER_CONCURRENCY=5

docker build -t pex-cache-write-fail-repro .

mkdir -p $PWD/pex-root
sudo find $PWD/pex-root/ -type f -name "*.lck" | while read lock_file; do
    # pex-root/downloads/2/.b727414169a36b7d524c1c3e31839a521725078d7b2ff038656844266160a992.atomic_directory.lck
    base_dir="$(dirname "${lock_file}")"
    lock_file_name="$(basename "${lock_file}")"
    target_dir_name="$(echo "${lock_file_name}" | cut -d. -f2)"
    sudo rm -rf "${base_dir}/${target_dir_name}"
done

declare -a pids
for ((x=0; x<CONTAINER_CONCURRENCY; x++)); do
    docker run --rm -v $PWD/pex-root:/tmp/pex-root pex-cache-write-fail-repro:latest repro.sh &
    pids+=($!)
done

for pid in "${pids[@]}"; do
    wait $pid
done

Yeah I mean that's our build machine setup - which is more annoying to repro when the dev machine is a mac.

I will beat my dead horse - its a brain sickness that devs who work on prod Linux apps are given macs.

I may show up to work on Monday and just walk over to IT and requisition something I can run ubuntu on; at least to repro this.

Yup! I used to wipe my IT supplied mac and install Linux, but now that intel is gone and M2 is old - no longer an option.

Yeah I loved my 2014 macbook air (with linux on it)

Make some noise! Get those macs recycled! Yeah - unfortunately.

OK it has reproduced itself in prod and since we've enabled the otel tracing of pants i now have the raw pex command that pants ran

Ok. If you can provide that the info can't hurt. Presumably the command runs fine in isolation though.

Yeah just removing some internal stuff before I post it here publicly

Ok. This is good to see but doesn't shed light.

Since we're using the lockfile we should never be able to download and then use a bad wheel right? Cause pex/pex via pip will checksum that against the sha in the lockfile?

Correct

The named caches are currently being written to a mount that's shared with the host. however there is only 1 build container running on the host at a time right now. However that build container's pants execution may run many pex-clis at once.

As does a normal Pants. That bit is not novel.

sorry the latter, we mount in the buildkite 'work' directory; presently the named caches are there. in .pants.d/named_caches

I'm not sure what to say. I sort of need a repro in hand. Code inspection + thinking through your setup seems to reveal no crease for write corruption.

Yeah it's late here. I think i'm going to put this down for now and sleep on it. We may try to run with some variant of Aidan's patch to get a clearer-error message when the corruption occurs to make it easier to catch 'in action'. If we did that is there anything in particular that would be useful to you to have? (since it seems to be many hours between occurrences in prod)

The read repair side can't provide useful info. The damage is done.

I was thinking to run that verification on the produced pex at build time

A patch like that would have hid that old Pants bug for example.

ie build it; then try to re-read it

But that still doesn't tell you what caused the bad write when you get 👎

no - but part of my problem there is that I have yet to be able to get on an actual host where the bad build happened

Gotcha

(Confirming for Jonathan since I'm working on this with him) - the instances do not use multi-attach, I checked the ASG instances with `aws ec2 describe-volumes`:

"Iops": 6000,
            "VolumeType": "gp3",
            "MultiAttachEnabled": false,

Ok. Thanks for checking @echoing-xylophone-39473.

Thanks for creating the

v2.101.2

Pex update, we're testing this out and will update by tomorrow

Any insights / new data from either the v2.101.2 Pex update or @echoing-psychiatrist-92346's change?

We initially got bit by the

LAST_ACCESS_FILE

write since we run the app in a container with a RO filesystem, so had to roll back / re-roll out. I believe the re-rollout is happening ~now unsure if we've had enough builds go through to hit a repro yet.

I'll ignore that because its unsettling.

LAST_ACCESS_FILE

is old news for Pex; so how was this something new you hit?

Not sure @echoing-xylophone-39473 / @echoing-psychiatrist-92346 would have more info there

I.E. ~2 years old:

:; git grep LAST_ACCESS_FILE
pex/cache/access.py:LAST_ACCESS_FILE = ".last-access"
pex/cache/access.py:    return os.path.join(pex_dir.path, LAST_ACCESS_FILE)
pex/venv/installer.py:                cache_access.LAST_ACCESS_FILE,

:; git blame pex/cache/access.py | grep LAST_ACCESS_FILE
991883c71 (John Sirois 2024-11-04 16:03:58 -0800 101) LAST_ACCESS_FILE = ".last-access"
991883c71 (John Sirois 2024-11-04 16:03:58 -0800 106)     return os.path.join(pex_dir.path, LAST_ACCESS_FILE)

:; git blame -- pex/venv/installer.py | grep LAST_ACCESS_FILE
991883c71 pex/venv/installer.py (John Sirois  2024-11-04 16:03:58 -0800 582)

The jump to v2.101.2 caused CrashLoopBackOff on read-only rootfs

File "/bin/app/pex", line 336, in boot
    with open(os.path.join(os.path.dirname(__file__), ".last-access"), "a") as fp:
OSError: [Errno 30] Read-only file system: '/bin/app/.last-access'

which I linked to pex 2.98.3 which added an unconditional last-access write to the generated

--venv

boot script with no try/except: github.com/pex-tool/pex/pull/3226 That collides with two things on our side: • we materialize the venv at build time into

/bin/app

, i.e. on the container root filesystem, not under

PEX_ROOT

• appfile-cli (custom tool which generates our kube manifests) sets

ReadOnlyRootFilesystem: true

unconditionally for every rendered container, with no appfile knob to opt out So the boot script tries to create

/bin/app/.last-access

on a read-only filesystem and dies before reaching the entrypoint. Worth noting the write buys us nothing anyway: it exists to feed

pex3 cache prune

, which only ever manages venvs inside

PEX_ROOT

. A build-time venv at

/bin/app

is never pruned. So I set

_PEX_CACHE_SET_LAST_ACCESS=0

in all container env which skips the write and app boots clean. I was also able to reproduced the crash and the fix locally.

we materialize the venv at build time into

/bin/app

Via what?

PEX_TOOLS=1 /bin/app venv ...

If so then the

.last-access

touch is a bug.

Yep, sorry should have been clearer, I rigged up the pipeline to emit some metrics for how often we are seeing this based on the diagnostics, that's what I'm waiting to permeate

Ok, the

.last-access

fix is available here: https://github.com/pex-tool/pex/releases/tag/v2.101.3

Enjoy! we'll def report back once we have some more info.

New data point from CI running pex 2.101.2, all of the failed CI runs we're looking at are docker image builds, and they split into two patterns. Some of the failed runs show the diagnostics firing exactly as designed: the `atomic_directory.py:290`/`:301` pair,

EEXIST

on

.lck.work

, then "Continuing to forcibly re-create the work directory" all across several different

downloads/

cache keys. In every case that's the only pair that appears; neither the "Failed to forcibly re-create" nor "Using new random workdir instead" follow-ups fired, meaning

rmtree

+

mkdir

succeeded cleanly on the first retry. Self-heal worked as intended in the runs that ultimately passed. Where the run still failed overall despite this pattern, the log tail cuts off before showing the actual terminal error, so I can't yet say whether that failure is downstream of this same event or unrelated Exact warnings, representative of this group:

[WARN] .../pex_root/installed_wheels/.../pex-2.101.2-py2.py3-none-any.whl/pex/atomic_directory.py:290:
  PEXWarning: [pid:<pid>, tid:<tid>, cwd:/tmp/pants-sandbox-<x>]: After obtaining an exclusive lock on
  .../pex_root/downloads/2/.<hash-a>.atomic_directory.lck,
  failed to establish a work directory at
  .../pex_root/downloads/2/<hash-a>.lck.work due to:
  [Errno 17] File exists: '.../downloads/2/<hash-a>.lck.work'

.../pex/atomic_directory.py:301: PEXWarning: [pid:<pid>, tid:<tid>, cwd:/tmp/pants-sandbox-<x>]:
  Continuing to forcibly re-create the work directory at
  .../pex_root/downloads/2/<hash-a>.lck.work.

.../pex/atomic_directory.py:290: PEXWarning: [pid:<pid>, tid:<tid2>, cwd:/tmp/pants-sandbox-<x>]: After
  obtaining an exclusive lock on
  .../pex_root/downloads/2/.<hash-b>.atomic_directory.lck,
  failed to establish a work directory at
  .../pex_root/downloads/2/<hash-b>.lck.work due to:
  [Errno 17] File exists: '.../downloads/2/<hash-b>.lck.work'

.../pex/atomic_directory.py:301: PEXWarning: [pid:<pid>, tid:<tid2>, cwd:/tmp/pants-sandbox-<x>]:
  Continuing to forcibly re-create the work directory at
  .../pex_root/downloads/2/<hash-b>.lck.work.

Same pid across both keys within a run, different tids which is consistent with your abnormal-termination read rather than a live lock race, same pattern you'd already established. Every run in this group shows the identical two-line shape, just different pids and cache keys, nothing new, just repeated independent occurrences. The other failed runs in this batch show a different pattern entirely: the exact symptom class from your earlier thread which was

ModuleNotFoundError: No module named 'pex.version'

on a

.bootstrap

zip during

venv --scope=deps

inside the Docker build but grepping the entire job log (and its sibling arch build in the same run) for `PEXWarning`/`atomic_directory.py` turns up zero matches. No EEXIST, no forcibly-recreate, nothing. CI's own retry-with-scratch-PEX_ROOT logic fired more than once per run and the corruption reproduced identically every time. Exact error, representative of this group:

#9 [deps 3/3] RUN PEX_TOOLS=1 /usr/local/bin/python3 /binary-deps.pex venv --scope=deps --compile /bin/app
#9 0.246 Traceback (most recent call last):
#9 0.246   File "<frozen runpy>", line 198, in _run_module_as_main
#9 0.246   File "<frozen runpy>", line 88, in _run_code
#9 0.246   File "/binary-deps.pex/__main__.py", line 280, in <module>
#9 0.246     result, should_exit, is_globals = boot(
#9 0.246                                       ^^^^^
#9 0.246   File "/binary-deps.pex/__main__.py", line 247, in boot
#9 0.246     from pex.variables import ENV, Variables
#9 0.246   File "/binary-deps.pex/.bootstrap/pex/variables.py", line 18, in <module>
#9 0.246   File "/binary-deps.pex/.bootstrap/pex/common.py", line 26, in <module>
#9 0.246   File "/binary-deps.pex/.bootstrap/pex/enum.py", line 13, in <module>
#9 0.246     'global_flag_repr', 'global_enum_repr', 'global_str', 'global_enum',
#9 0.246 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#9 0.246   File "/binary-deps.pex/.bootstrap/pex/exceptions.py", line 12, in <module>
#9 0.246 ModuleNotFoundError: No module named 'pex.version'
#9 ERROR: process "/bin/sh -c PEX_TOOLS=1 /usr/local/bin/python3 /binary-deps.pex venv --scope=deps --compile /bin/app"
    did not complete successfully: exit code: 1

A

.bootstrap

chroot missing

pex/version.py

in every run in this group, same failure signature as the earlier builds you and I discussed, reproduced across multiple retries against a fresh scratch

PEX_ROOT

before the run gave up. That's exactly the shape your "second mechanism" concern predicts: a corrupted artifact with the identical downstream symptom, but no trace in the loud

atomic_directory

EEXIST path #3263 instruments which is consistent with the

pep_427.py:1119-1131

skip-on-exists copies (

safe_copy(overwrite=False)

,

elif not os.path.exists(dst_file)

, the symlink-EEXIST swallow) quietly preserving stale content into a populate that never trips `os.mkdir`'s EEXIST check in the first place, so there's nothing for #3263 to warn about.

Aiden, do we see any logs from pants indicating that per got terminated outside of its control?

Yea I think so, and we see these two logs arrive at the same time which is kinda sus

20:07:14.84 [INFO] Canceled: Building 79 requirements for .../migration/alembic_pex.pex from the
  3rdparty/python/deps_lock.txt resolve: PyMySQL, aiohttp, alembic, anthropic, ... (18.3s)
20:07:14.84 [WARN] .../pex/atomic_directory.py:290: PEXWarning: [pid:<pid>, ...]: After obtaining
  an exclusive lock on ..., failed to establish a work directory at ... due to: [Errno 17] File exists

Pants is killing one pex build which lines up exactly with another pex process tripping over a leftover .lck.work dir. Also Pants canceled a pex build very shortly into building it, and that's the exact pex that shows up corrupted later in the same run.

Yeah those pants cancellations are absolutely expected for how pants does things (its management of compute + hedging remote cache requests). But that's not the same as a pex getting OOM killed out from under pants for example.

It is the same from Pex's perspective and neither method of kill breaks the assumptions of atomic_directory which is flock + atomic rename or nothing.

also that 2.101.3 readonly patch is working great

Unfortunately, climbing cancelled.

yea, not really, weve put a mitigation in place to disable the remote cache on retries though (not that thats a solution)

So, focusing on docker image builds. That's via docker in docker. Has that fact changed recently?

Its DinD but by mounting the socket

Same question remains. Is that new?

Nope.

K

Yeah there's not a ton to go on 😞

So ... if I were an employee with access I'd be eliminating the dnd and just running, say, tests that had no dep on a docker image being built.

Ok, I was able to pull a couple poisoned pex's from the cache and they seem to point to

os.walk

failing silently.

.deps/mypy-1.8.0-…whl

blew up but is a perfectly valid zip with 2 members:

.layout.json

plus one

.so

where the healthy packed wheel has 1300, including 81 directory entries. The poisoned one has zero, i.e. the walk never descended past the chroot root. No dist-info/METADATA`, which is what

PEX_TOOLS=1 … venv

chokes on.

deterministic_walk

(

pex/common.py

) wraps

os.walk

and no call site passes

onerror

, so a failed directory listing gets discarded and that subtree is silently omitted then the walk then finishes "successfully" with a strict subset of the tree.

PEXBuilder._add_dist

registers only the chroot's root-level files,

_build_packedapp

zips that subset, and

atomic_directory

takes its success path and finalizes it.

_add_dist

returns the fingerprint of the complete chroot, not a hash of what it actually collected so the truncated wheel gets cached under a key asserting it's whole. The fix is small github.com/pex-tool/pex/pull/3269 default

onerror

to re-raise so we fail at the listing instead of shipping a broken PEX. Reproduced it against pex's own builder and added regression tests. tbh it is still slightly unknown what makes these fail on our build agents, my best guess is fd exhaustion from the parallel docker builds or ENOENT from concurrent cache mutation.

Excellent. That explanation holds water. I will be interested to see what raising on walk error reveals about your CI setup! Please don't wait on me - out climbing. Rip your own release with

uv run dev-cmd package

and report back!

For a quick update, I built the package and deployed it out, we have not seen any instances of the poisoning since, im going to continue letting it percolate for the rest of the day though then collect some hard numbers

Ok. But you should see hard errors right? That's the trade, no slow poison error, a quick hard error.

yep and it should fail at

pants package

on the pex build now instead of 20 min later at the docker build stage. I'm only sampling our main pipeline so far and haven't caught one there yet. Pulling data across all branches/pipelines later today for a real sample. I put in a guard yesterday as well to detect and evict incomplete cache entries preflight which has been helping as well to clean out the poison, but this should make it so no incomplete packages get in in the first place

Alright @echoing-psychiatrist-92346 your

os.walk

fail fast fix is now available here: https://github.com/pex-tool/pex/releases/tag/v2.101.4

I took a look over the long weekend + fri and I think the proactive check + cleanup mechanisms we put in place to stabilize builds for folks are preventing this error from being hit right now. @echoing-psychiatrist-92346 would be in a better spot to confirm though since he did the majority of that work.

Thanks for the report. That's definitely very disappointing from a debug perspective. Stability is stability though, so it's good you have that back.

Yeah I think with this hard error now in place unwinding some of those mitigations is now safe (ie we avoid most cache poisoning scenarios). so we can 👀 that.

sorry, was out for the extended weekend so just catching back up, but yea like jonathan said we have been catching and remediating the bad caches now automatically through our ci pipelines, but you can see the decline in stage failures starting with the mitigation, then with the implementation of the updated pex version (to further limit the bad pex from making it to the remote cache)

I will be very interested to learn what the now hidden errors are. It seems likely some other user is doing whatever you're doing in CI to trigger directory walk errors, which seem overwhelmingly likely to be the culprit here (the only other known is a file system setup that invalidates flock / atomic directory rename assumptions - and you all have ruled that out).

let me rig somthing up to try and capture / alert on these in our pipelines

Thanks. I assume this would be interesting to you all as well. Disk-fulls, ulimits too low, weird file perms ... something like that must be happening in CI.