Using pants 2 26 0 I m seeing an error when trying to run `p Pants #general

Using pants 2.26.0, I'm seeing an error when tryin...

elegant-florist-94385

08/13/2025, 2:50 PM

Using pants 2.26.0, I'm seeing an error when trying to run

pants export

. My venv has already been generated, but I'm trying to upgrade an existing package. Generate lockfiles works fine, but I can't export the venv after. It gives me:

PermissionError: [Errno 13] Permission denied: '/home/luke/.cache/pants/named_caches/pex_root/installed_wheels/0/6f2a054e82a173d41ffc60f43ca40e828b28894f76639f69797ecfed44fcca9b/faststream-0.6.0rc0-py3-none-any.whl.lck.work/faststream/__about__.py'

Has anyone seen something like this before? (stacktrace in 🧵 )

elegant-florist-94385

08/13/2025, 2:50 PM

Copy code

$ pants export --py-resolve-format=mutable_virtualenv --resolve=universal_resolve
08:47:27.61 [WARN] A plugin is calling `await Effect(InteractiveProcessResult, InteractiveProcess, process)` directly. This will cause restarting logic not to be applied. Use `await run_interactive_process(process)` or `await run_interactive_process_in_environment(process, environment_name)` instead. See <https://github.com/pantsbuild/pants/blob/release_2.26.0/src/python/pants/engine/intrinsics.py> for more details.
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/luke/.cache/pants/named_caches/pex_root/unzipped_pexes/1/30c87eeb6cbfeeb90a389a9e2d70ad5089858d75/__main__.py", line 236, in <module>
    result, should_exit, is_globals = boot(
                                      ^^^^^
  File "/home/luke/.cache/pants/named_caches/pex_root/unzipped_pexes/1/30c87eeb6cbfeeb90a389a9e2d70ad5089858d75/__main__.py", line 228, in boot
    result = bootstrap_pex(
             ^^^^^^^^^^^^^^
  File "/home/luke/.cache/pants/named_caches/pex_root/unzipped_pexes/1/30c87eeb6cbfeeb90a389a9e2d70ad5089858d75/.bootstrap/pex/pex_bootstrapper.py", line 680, in bootstrap_pex
    return pex.PEX(entry_point).execute()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/luke/.cache/pants/named_caches/pex_root/unzipped_pexes/1/30c87eeb6cbfeeb90a389a9e2d70ad5089858d75/.bootstrap/pex/pex.py", line 579, in execute
    result = self._wrap_coverage(self._wrap_profiling, self._execute)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/luke/.cache/pants/named_caches/pex_root/unzipped_pexes/1/30c87eeb6cbfeeb90a389a9e2d70ad5089858d75/.bootstrap/pex/pex.py", line 486, in _wrap_coverage
    return runner(*args)
           ^^^^^^^^^^^^^
  File "/home/luke/.cache/pants/named_caches/pex_root/unzipped_pexes/1/30c87eeb6cbfeeb90a389a9e2d70ad5089858d75/.bootstrap/pex/pex.py", line 517, in _wrap_profiling
    return runner(*args)
           ^^^^^^^^^^^^^
  File "/home/luke/.cache/pants/named_caches/pex_root/unzipped_pexes/1/30c87eeb6cbfeeb90a389a9e2d70ad5089858d75/.bootstrap/pex/pex.py", line 618, in _execute
    return self.execute_entry(parse_entry_point(self._pex_info_overrides.entry_point))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/luke/.cache/pants/named_caches/pex_root/unzipped_pexes/1/30c87eeb6cbfeeb90a389a9e2d70ad5089858d75/.bootstrap/pex/pex.py", line 796, in execute_entry
    return self.execute_module(entry_point.module)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/luke/.cache/pants/named_caches/pex_root/unzipped_pexes/1/30c87eeb6cbfeeb90a389a9e2d70ad5089858d75/.bootstrap/pex/pex.py", line 804, in execute_module
    return Globals(runpy.run_module(module_name, run_name="__main__", alter_sys=True))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen runpy>", line 226, in run_module
  File "<frozen runpy>", line 98, in _run_module_code
  File "<frozen runpy>", line 88, in _run_code
  File "/home/luke/.cache/pants/named_caches/pex_root/installed_wheels/0/0f328a89a0aec3dfc0112a39103fc5f45253c30f959c5db4da5260e607dd4a80/pex-2.33.4-py2.py3-none-any.whl/pex/tools/__main__.py", line 10, in <module>
    sys.exit(tools())
             ^^^^^^^
  File "/home/luke/.cache/pants/named_caches/pex_root/installed_wheels/0/0f328a89a0aec3dfc0112a39103fc5f45253c30f959c5db4da5260e607dd4a80/pex-2.33.4-py2.py3-none-any.whl/pex/tools/main.py", line 72, in main
    interpreter = pex_bootstrapper.find_compatible_interpreter(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/luke/.cache/pants/named_caches/pex_root/installed_wheels/0/0f328a89a0aec3dfc0112a39103fc5f45253c30f959c5db4da5260e607dd4a80/pex-2.33.4-py2.py3-none-any.whl/pex/pex_bootstrapper.py", line 321, in find_compatible_interpreter
    target = _select_path_interpreter(
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/luke/.cache/pants/named_caches/pex_root/installed_wheels/0/0f328a89a0aec3dfc0112a39103fc5f45253c30f959c5db4da5260e607dd4a80/pex-2.33.4-py2.py3-none-any.whl/pex/pex_bootstrapper.py", line 233, in _select_path_interpreter
    for interpreter in candidate_interpreters_iter:
  File "/home/luke/.cache/pants/named_caches/pex_root/installed_wheels/0/0f328a89a0aec3dfc0112a39103fc5f45253c30f959c5db4da5260e607dd4a80/pex-2.33.4-py2.py3-none-any.whl/pex/pex_bootstrapper.py", line 189, in iter_compatible_interpreters
    valid_or_error = _valid_interpreter(interpreter)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/luke/.cache/pants/named_caches/pex_root/installed_wheels/0/0f328a89a0aec3dfc0112a39103fc5f45253c30f959c5db4da5260e607dd4a80/pex-2.33.4-py2.py3-none-any.whl/pex/pex_bootstrapper.py", line 167, in _valid_interpreter
    return interpreter_test.test_resolve(interp) if interpreter_test else True
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/luke/.cache/pants/named_caches/pex_root/installed_wheels/0/0f328a89a0aec3dfc0112a39103fc5f45253c30f959c5db4da5260e607dd4a80/pex-2.33.4-py2.py3-none-any.whl/pex/pex_bootstrapper.py", line 82, in test_resolve
    pex_environment = PEXEnvironment.mount(
                      ^^^^^^^^^^^^^^^^^^^^^
  File "/home/luke/.cache/pants/named_caches/pex_root/installed_wheels/0/0f328a89a0aec3dfc0112a39103fc5f45253c30f959c5db4da5260e607dd4a80/pex-2.33.4-py2.py3-none-any.whl/pex/environment.py", line 250, in mount
    installed_pex = ensure_installed(pex=pex, pex_root=pex_root, pex_hash=pex_hash)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/luke/.cache/pants/named_caches/pex_root/installed_wheels/0/0f328a89a0aec3dfc0112a39103fc5f45253c30f959c5db4da5260e607dd4a80/pex-2.33.4-py2.py3-none-any.whl/pex/layout.py", line 704, in ensure_installed
    return _ensure_installed(layout, pex_root, pex_hash)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/luke/.cache/pants/named_caches/pex_root/installed_wheels/0/0f328a89a0aec3dfc0112a39103fc5f45253c30f959c5db4da5260e607dd4a80/pex-2.33.4-py2.py3-none-any.whl/pex/layout.py", line 362, in _ensure_installed
    _ensure_distributions_installed(
  File "/home/luke/.cache/pants/named_caches/pex_root/installed_wheels/0/0f328a89a0aec3dfc0112a39103fc5f45253c30f959c5db4da5260e607dd4a80/pex-2.33.4-py2.py3-none-any.whl/pex/layout.py", line 294, in _ensure_distributions_installed
    _ensure_distributions_installed_serial(
  File "/home/luke/.cache/pants/named_caches/pex_root/installed_wheels/0/0f328a89a0aec3dfc0112a39103fc5f45253c30f959c5db4da5260e607dd4a80/pex-2.33.4-py2.py3-none-any.whl/pex/layout.py", line 211, in _ensure_distributions_installed_serial
    _install_distribution(item, layout, pex_info, work_dir, install_to)
  File "/home/luke/.cache/pants/named_caches/pex_root/installed_wheels/0/0f328a89a0aec3dfc0112a39103fc5f45253c30f959c5db4da5260e607dd4a80/pex-2.33.4-py2.py3-none-any.whl/pex/layout.py", line 190, in _install_distribution
    layout.extract_dist(
  File "/home/luke/.cache/pants/named_caches/pex_root/installed_wheels/0/0f328a89a0aec3dfc0112a39103fc5f45253c30f959c5db4da5260e607dd4a80/pex-2.33.4-py2.py3-none-any.whl/pex/layout.py", line 551, in extract_dist
    install_wheel_chroot(dist_path, dest_dir)
  File "/home/luke/.cache/pants/named_caches/pex_root/installed_wheels/0/0f328a89a0aec3dfc0112a39103fc5f45253c30f959c5db4da5260e607dd4a80/pex-2.33.4-py2.py3-none-any.whl/pex/pep_427.py", line 128, in install_wheel_chroot
    wheel = install_wheel(
            ^^^^^^^^^^^^^^
  File "/home/luke/.cache/pants/named_caches/pex_root/installed_wheels/0/0f328a89a0aec3dfc0112a39103fc5f45253c30f959c5db4da5260e607dd4a80/pex-2.33.4-py2.py3-none-any.whl/pex/pep_427.py", line 213, in install_wheel
    record_files(
  File "/home/luke/.cache/pants/named_caches/pex_root/installed_wheels/0/0f328a89a0aec3dfc0112a39103fc5f45253c30f959c5db4da5260e607dd4a80/pex-2.33.4-py2.py3-none-any.whl/pex/pep_427.py", line 204, in record_files
    InstalledWheel.create_installed_file(path=file_abspath, dest_dir=dest)
  File "/home/luke/.cache/pants/named_caches/pex_root/installed_wheels/0/0f328a89a0aec3dfc0112a39103fc5f45253c30f959c5db4da5260e607dd4a80/pex-2.33.4-py2.py3-none-any.whl/pex/pep_376.py", line 267, in create_installed_file
    hashing.file_hash(path, digest=hasher)
  File "/home/luke/.cache/pants/named_caches/pex_root/installed_wheels/0/0f328a89a0aec3dfc0112a39103fc5f45253c30f959c5db4da5260e607dd4a80/pex-2.33.4-py2.py3-none-any.whl/pex/hashing.py", line 185, in file_hash
    with open(path, "rb") as fp:
         ^^^^^^^^^^^^^^^^
PermissionError: [Errno 13] Permission denied: '/home/luke/.cache/pants/named_caches/pex_root/installed_wheels/0/6f2a054e82a173d41ffc60f43ca40e828b28894f76639f69797ecfed44fcca9b/faststream-0.6.0rc0-py3-none-any.whl.lck.work/faststream/__about__.py'
08:47:29.02 [ERROR] 1 Exception encountered:

Engine traceback:
  in root
    ..
  in pants.core.goals.export.export
    `export` goal

Traceback (most recent call last):
  File "/home/luke/.cache/nce/830340ce637c599fc91aa9134dedaf6fcc4aa28cff14aaee2d17a9fb389f3788/bindings/venvs/2.26.0/lib/python3.11/site-packages/pants/core/goals/export.py", line 223, in export
    raise ExportError(f"Failed to write {result.description} to {result_dir}")
pants.core.goals.export.ExportError: Failed to write mutable virtualenv for universal_resolve (using Python 3.12.3) to dist/export/python/virtualenvs/universal_resolve/3.12.3

elegant-florist-94385

08/13/2025, 3:06 PM

For the record, I can install this version of the library using pip directly from my (pants-generated) venv. eg.

./dist/export/python/virtualenvs/universal_resolve/3.12.3/bin/pip3 install faststream==0.6.0rc0

works fine

fast-nail-55400

08/13/2025, 3:08 PM

Can you retry with Pants configured to use the latest Pex?

fast-nail-55400

08/13/2025, 3:10 PM

Copy code

[pex-cli]
version = "v2.50.2"
known_versions = [
"v2.50.2|macos_arm64|b787124d85ad26be62e64e0d2a4f6e5ecedb9b6c83e64dbcce835ccc63a1d392|4772270", "v2.50.2|macos_x86_64|b787124d85ad26be62e64e0d2a4f6e5ecedb9b6c83e64dbcce835ccc63a1d392|4772270",  "v2.50.2|linux_x86_64|b787124d85ad26be62e64e0d2a4f6e5ecedb9b6c83e64dbcce835ccc63a1d392|4772270",  "v2.50.2|linux_arm64|b787124d85ad26be62e64e0d2a4f6e5ecedb9b6c83e64dbcce835ccc63a1d392|4772270",
]

fast-nail-55400

08/13/2025, 3:11 PM

(Pants relies on Pex for most Python venv tasks.)

elegant-florist-94385

08/13/2025, 3:11 PM

Trying it now

elegant-florist-94385

08/13/2025, 3:13 PM

Unfortunately it still gives me the same error

fast-nail-55400

08/13/2025, 3:13 PM

Okay, are you able to reproduce the issue with just a Pex invocation (and take Pants out of the equation)?

fast-nail-55400

08/13/2025, 3:14 PM

(use

--keep-sandboxes

to get the failing execution sandbox and the pex invocation will be in the

__run.sh

script)

fast-nail-55400

08/13/2025, 3:14 PM

The goal being to narrow down whether Pex is having an issue with this particular package, and, if so, then report upstream to Pex.

elegant-florist-94385

08/13/2025, 4:34 PM

If I just run the

__run.sh

directly, it seems to work.

Copy code

luke@luke-dev:/tmp/pants-sandbox-gGFS6E$ cat __run.sh 
#!/usr/bin/env bash
# This command line should execute the same process as pants did internally.
cd /tmp/pants-sandbox-gGFS6E
env -i CPPFLAGS='' HTTPS_PROXY=$'<http://PITC-Zscaler-Americas-Cincinnati3PR.proxy.corporate.ge.com:80>' HTTP_PROXY=$'<http://PITC-Zscaler-Americas-Cincinnati3PR.proxy.corporate.ge.com:80>' LANG=en_US.UTF-8 LDFLAGS='' NO_PROXY=localhost,127.0.0.1,ge.com,av.ge.com,build.ge.com,gecdn.com PATH=$'/home/luke/.local/bin:/home/luke/.nvm/versions/node/v20.12.2/bin:/home/luke/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin' PEX_IGNORE_RCFILES=true PEX_PYTHON=/home/luke/.cache/nce/830340ce637c599fc91aa9134dedaf6fcc4aa28cff14aaee2d17a9fb389f3788/bindings/venvs/2.26.0/bin/python3.11 PEX_ROOT=.cache/pex_root http_proxy=$'<http://PITC-Zscaler-Americas-Cincinnati3PR.proxy.corporate.ge.com:80>' https_proxy=$'<http://PITC-Zscaler-Americas-Cincinnati3PR.proxy.corporate.ge.com:80>' no_proxy=localhost,127.0.0.1,ge.com,av.ge.com,build.ge.com,gecdn.com /home/luke/.cache/nce/830340ce637c599fc91aa9134dedaf6fcc4aa28cff14aaee2d17a9fb389f3788/bindings/venvs/2.26.0/bin/python3.11 ./pex --tmpdir .tmp --no-emit-warnings --pip-version 24.2 --cert ca-certificates.crt --python-path /usr/bin --interpreter-constraint $'CPython==3.12.*' -- -c $'import hashlib, os, sys\n\npython = os.path.realpath(sys.executable)\nprint(python)\n\nhasher = hashlib.sha256()\nwith open(python, "rb") as fp:\n  for chunk in iter(lambda: fp.read(8192), b""):\n      hasher.update(chunk)\nprint(hasher.hexdigest())\n'

Copy code

luke@luke-dev:/tmp/pants-sandbox-gGFS6E$ ./__run.sh
/usr/bin/python3.12
2845674b3fae665f5f6c3f01674fefa4e495f15f5d755ee8f7f1a40575abdc8a
luke@luke-dev:/tmp/pants-sandbox-gGFS6E$

fast-nail-55400

08/13/2025, 5:08 PM

What about the permissions on the path which ostensibly triggers the error?

fast-nail-55400

08/13/2025, 5:08 PM

And drastic but does wiping the names caches tree “fix” the issue?

fast-nail-55400

08/13/2025, 5:09 PM

(In case there was some sort of cache corruption.)

elegant-florist-94385

08/13/2025, 5:10 PM

I was just looking into that now:

Copy code

luke@luke-dev:~/.cache/pants/named_caches/pex_root/installed_wheels/0/6f2a054e82a173d41ffc60f43ca40e828b28894f76639f69797ecfed44fcca9b$ ls -al
total 48
drwxrwxr-x   2 luke luke  4096 Aug 13 11:02 .
drwxrwxr-x 265 luke luke 40960 Aug 13 09:11 ..
-rwxrwxr-x   1 luke luke     0 Aug 13 11:02 .faststream-0.6.0rc0-py3-none-any.whl.atomic_directory.lck

That path apparently does not exist. Only an atomic_directory lock. (I'm assuming some kind of tarball gets unpacked there but is then cleaned up on exiting)

elegant-florist-94385

08/13/2025, 5:11 PM

How do I wipe the named caches? Just

rm -rf ~/.cache/pants/named_caches

👍 1

elegant-florist-94385

08/13/2025, 5:24 PM

Unfortunately, its the same issue

elegant-florist-94385

08/13/2025, 5:35 PM

Went and found the faststream whl download in the

named_caches/downloads

directory. I unzipped it and apparently all the files inside are only writable?:

Copy code

luke@luke-dev:~/temp/faststream$ ls -al
total 108
drwxr-xr-x 16 luke luke 4096 Jan  1  1980 .
drwxrwxr-x  4 luke luke 4096 Aug 13 11:32 ..
--w--wx---  1 luke luke  200 Jan  1  1980 __about__.py
--w--wx---  1 luke luke  332 Jan  1  1980 annotations.py
--w--wx---  1 luke luke 3812 Jan  1  1980 app.py
drwxr-xr-x  2 luke luke 4096 Jan  1  1980 asgi
--w--wx---  1 luke luke   66 Jan  1  1980 cli.py
drwxr-xr-x 11 luke luke 4096 Jan  1  1980 confluent
--w--wx---  1 luke luke 5714 Jan  1  1980 exceptions.py
--w--wx---  1 luke luke  974 Jan  1  1980 __init__.py
drwxr-xr-x 12 luke luke 4096 Jan  1  1980 _internal
drwxr-xr-x 11 luke luke 4096 Jan  1  1980 kafka
--w--wx---  1 luke luke  607 Jan  1  1980 __main__.py
drwxr-xr-x  2 luke luke 4096 Jan  1  1980 message
drwxr-xr-x  3 luke luke 4096 Jan  1  1980 middlewares
drwxr-xr-x 11 luke luke 4096 Jan  1  1980 nats
drwxr-xr-x  2 luke luke 4096 Jan  1  1980 opentelemetry
drwxr-xr-x  2 luke luke 4096 Jan  1  1980 params
drwxr-xr-x  2 luke luke 4096 Jan  1  1980 prometheus
--w--wx---  1 luke luke    0 Jan  1  1980 py.typed
drwxr-xr-x 11 luke luke 4096 Jan  1  1980 rabbit
drwxr-xr-x 11 luke luke 4096 Jan  1  1980 redis
drwxr-xr-x  2 luke luke 4096 Jan  1  1980 response
--w--wx---  1 luke luke 5452 Jan  1  1980 security.py
drwxr-xr-x  5 luke luke 4096 Jan  1  1980 specification
--w--wx---  1 luke luke  627 Jan  1  1980 types.py

fast-nail-55400

08/13/2025, 6:24 PM

yeah that is weird

fast-nail-55400

08/13/2025, 6:25 PM

is the wheel on PyPI that way?

fast-nail-55400

08/13/2025, 6:25 PM

also, the exception being thrown is at https://github.com/pantsbuild/pants/blob/7de74f97be7963469f28847b74453a84db93f2de/src/python/pants/core/goals/export.py#L231

fast-nail-55400

08/13/2025, 6:26 PM

some more logging to capture the exact command being run at that point via

InteractiveProcess

would be helpful

fast-nail-55400

08/13/2025, 6:26 PM

I believe the sandbox you looked at is just the one which fingerprints the Python binary

fast-nail-55400

08/13/2025, 6:26 PM

and we may not make save sandboxes when

InteractiveProcess

is being used by rule code

fast-nail-55400

08/13/2025, 6:27 PM

so if you have the ability and wherewithal to run Pants from source, some additional logging in core/goals/export.py might help. narrow down the issue.

fast-nail-55400

08/13/2025, 6:27 PM

with a separate track for figuring out why the wheel is the way it is

elegant-florist-94385

08/13/2025, 6:44 PM

Looks like in faststream 0.5.48 (previous release), all the files are readable but not writable. I guess this somehow got reversed in the 0.6.0rc0 release. I'll go raise an issue on their repo. Seems it wasn't caught because (from my quick testing at least) pip and poetry are both happy enough with just having the files present.

brief-scientist-13682

08/13/2025, 6:59 PM

https://github.com/ag2ai/faststream/issues/2448

elegant-florist-94385

08/13/2025, 7:00 PM

I was on my way to go create the bug and just saw that. Thanks!

brief-scientist-13682

08/13/2025, 7:00 PM

It was right there in the original OP.

brief-scientist-13682

08/13/2025, 7:00 PM

Always check the obvious.

16 Views

Open in Slack

Previous Next