#20108 Running a `python_requirement` pex from store, after clearing named caches, fails with `<sandbox>/.../pex: No such file or directory` Issue created by huonw Describe the bug A

python_requirement

can be run directly, both via

pants run ...

and

adhoc_tool

. In either case, Pants builds a venv pex internally which can be cached (including the shim script!), and the runs it. Runs after the first can sometimes fail, because the "build" and "run" use slightly different env vars (

PEX_PYTHON

vs

PEX_PYTHON_PATH

), and the cached shim script hardcodes hashes from the "build" that are changed in the "run" due to those vars. 1. run 1: • builds the venv pex, including putting hashes into a shim script • stores that script (and everything else) in local/remote store • seeds the named cache, as a (desired) side-effect optimisation • runs the shim script with slightly different env vars, but the named cache is already seeded, so the look-ups happen fine 2. "clear" the named cache e.g. actively clear it manually with

rm

, or, by running on a new machine that uses the (remote) store but doesn't share the named caches (this is how we encountered it, in our CI) 3. run 2: • restores the venv pex files from cache (so the named cache seeding side-effect doesn't happen) • runs the shim script, and decides it needs to re-seed the named cache... but it's using different env vars this time, so what's seeded doesn't match the hashes in the shim script. Reproducer:

cd $(mktemp -d)

# dedicated dir to make it easier to control
cache=$(mktemp -d)

cat > pants.toml <<EOF
[GLOBAL]
pants_version = "2.17.0"
backend_packages = ["pants.backend.python"]
local_store_dir = "$cache/store"
named_caches_dir = "$cache/named"

[python]
interpreter_constraints = ["==3.9.*"]
EOF

echo 'python_requirement(name="cowsay", requirements=["cowsay"])' > BUILD


pants run :cowsay -- -t 1

# clear out the named cache, e.g. user doing clean up, or running on a fresh machine in CI
rm -rf "$cache/named"

# BUG: .../pex: No such file or directory
pants run :cowsay -- -t 2

Output:

22:39:41.60 [INFO] Initializing scheduler...
22:39:44.94 [INFO] Scheduler initialized.
22:39:50.10 [INFO] Starting: Building local_dists.pex
22:39:50.90 [INFO] Completed: Building local_dists.pex
22:39:51.00 [INFO] Starting: Building cowsay.pex with 1 requirement: cowsay
22:39:57.77 [INFO] Completed: Building cowsay.pex with 1 requirement: cowsay
  _
| 1 |
  =
 \
  \
    ^__^
    (oo)\_______
    (__)\       )\/\
        ||----w |
        ||     ||
/private/var/folders/sv/vd266m4d4lvctgs2wpnhjs9w0000gn/T/pants-sandbox-T8PpXp/./cowsay.pex_pex_shim.sh: line 53: /private/var/folders/sv/vd266m4d4lvctgs2wpnhjs9w0000gn/T/pants-sandbox-T8PpXp/./.cache/pex_root/venvs/02c88e4cdfff925552483c12515d5c61f48eec89/5c8d59a96928304149d8aa767541cee82acb72c3/pex: No such file or directory

Pants version 2.16.0, 2.17.0, 2.18.0rc3, 2.19.0.dev3 OS macOS, Linux Additional info This doesn't appear to apply running a

python_sources

or

pex_binary

, only

python_requirement

. pantsbuild/pants