Has anyone seen in a CI environment where building the

dockerfile_parser.pex

takes forever and never finishes?

14387.86s	Building dockerfile_parser.pex from <resource://pants.backend.docker.subsystems/dockerfile.lock>

That's a bit wild. Under the hood, PEX should be installing deps with exact versions from the internal lockfile, so it's not trapped solving for versions. Can you give us more details about the CI env? Also are you using a custom pypi mirror?

Custom pypi mirror, yes. Code Artifact. I tried with

-ldebug

and

--keep-sandboxes=always

and couldn't see anything interesting in the sandbox.

Does it stall when run from the sandbox? I'm wondering if a download is stalling or a middlebox is causing trouble. We've used a different mirror, but have had trouble where it would just drop connections (which caused failure rather than hanging)

The sandbox doesn't contain a __run.sh so I'm not sure how to interactively run whatever pants is doing.

actually, which Pants version are you using? We changed to a rust-based parser in 2.23, enabling that option that might just sidestep the issue

Yes, we just used that to workaround the issue 🙂

I'm glad we have a workaround. But there should definitely be a "__run.sh". So that's spooky.

> tree
.
├── dockerfile.lock
├── dockerfile_parser.pex
│   ├── __main__.py
│   ├── __pants_df_parser.py
│   ├── __pex__
│   │   └── __init__.py
│   ├── pex -> __main__.py
│   └── PEX-INFO
├── pex
├── __run.sh
└── source_files
    └── __pants_df_parser.py

yep, no __run.sh.

It's just everything but that? If we aren't writing it out, I haven't deeply looked into the codepath that generates it, but it shouldn't be doing anything fancy where we would stall. Maybe the PEX invocation? The "__run.sh" should contain something like the following:

./pex --output-file dockerfile_parser.pex --venv prepend --seed verbose --venv-site-packages-copies --python /usr/bin/python3.9 --entry-point __pants_df_parser --no-pre-install-wheels '--sources-directory=source_files' --lock dockerfile.lock --no-pypi '--index=<https://pypi.org/simple/>' --manylinux manylinux2014 --layout packed

can you try running that from inside the sandbox?

Yeah, I can try it tomorrow. I'll report back. Thanks!

That sounds genuinely spooky. What did you end up doing to fix it? Just purge caches/reimage runners?

I'll be honest, I'm not the one working on it. One of my team members is. I've been pretty far, but I think installing golang using their recommended installer instead of mise fixed it... I don't know why. My teammate believes it has something to do with symlinks maybe?

Hi team) I’m currently working on an issue involving pants package hanging during the Dockerfile parser PEX build on an Amazon Linux 2023 *ARM*64 environment. The command being run is:

pants --keep-sandboxes=always -ldebug --pex-verbosity=9 package base-images/fips-golang:docker-build-arm64

The process hangs at this log message:

Building dockerfile_parser.pex from <resource://pants.backend.docker.subsystems/dockerfile.lock>

When inspecting the sandbox prepared for this step, the structure looks like this:

.
├── dockerfile.lock
├── pex
└── source_files
    └── __pants_df_parser.py

According to the logs, the command being executed inside the sandbox is:

/home/ssm-user/.cache/nce/244e442a2bba37173b243362ceb6a425f667e7d22b86fab91862cdbfdc540ebb/bindings/venvs/2.24.1/bin/python3.9 ./pex \
  --tmpdir .tmp \
  --jobs 2 \
  -vvvvvvvvv \
  --no-emit-warnings \
  --pip-version 24.2 \
  --python-path "/home/ssm-user/.local/bin:/home/ssm-user/bin:/mise/shims:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin" \
  --output-file dockerfile_parser.pex \
  --venv prepend \
  --seed verbose \
  --venv-site-packages-copies \
  --python /usr/bin/python3.9 \
  --entry-point __pants_df_parser \
  --no-pre-install-wheels \
  --sources-directory=source_files \
  --lock dockerfile.lock \
  --no-pypi \
  --index=<https://aws>:<TOKEN>@altana-<ECR-AWS-ACCOUNT>.<http://d.codeartifact.us-east-1.amazonaws.com/pypi/altana/simple|d.codeartifact.us-east-1.amazonaws.com/pypi/altana/simple> \
  --manylinux manylinux2014 \
  --layout packed

If I run this exact command manually in the sandbox, it completes successfully and creates the dockerfile_parser.pex file. No errors are reported. However, when invoked via pants package, the process hangs and no additional logs seem to be generated. I’ve tried increasing verbosity, enabling sandbox preservation, etc., but I can’t find any useful clues in the logs. A strong suspect here is the Go runtime installed via mise. Specifically: • With Go 1.23.2 installed via mise → Pants hangs on this operation. • With Go 1.23.2 installed from raw binaries → the operation completes successfully. We would ideally like to keep using mise, so we’re trying to identify the real root cause here. Is there a way to enable deeper logging for whatever subprocess (possibly Go-related) is causing this hang? Or any recommendation on how to troubleshoot further? I have full sandbox snapshots and logs available and happy to dig into anything further you suggest. Thanks a lot in advance!

@careful-address-89803 to loop back here. It turns out it was being caused by mise shims. I don't know how the shim would cause headaches for pants, but it does. The solution was to stop using shims and add all bins to PATH. The reproduction is pretty simple, just add mise shims for python bin in your PATH.

I wonder if that has to do with Pants's hermeticity for envvars. Unless they're specifically called out to be forwarded, they aren't. It might also be that the binaries don't fingerprint properly? I haven't really dealt with either the golang backend or binary location. If you've got a reproducer repo it would be easier to look into.