It'll log the conserved path for each process, and in that dir the

__run.sh

script will execute the process

so you can tinker with that script as needed and re-run it directly inside that dir

That is often a really useful way to debug issues like this

That is a weird error

That

venv_dir = os.path.abspath(os.path.dirname(__file__))

line 7 is in the

__main__.py

that pex synthesizes to make its venv runnable.

And

syntax error near unexpected token

is a bash error

So bash is trying to execute that file as shell instead of as python

meaning that, probably, the shebang in that file isn't pointing to a valid python interpreter

So the next step would be to see what the shebang is for the failing process

I see on my workstation, these files are located in ~/.cache/pants/named_caches/pex_root, whereas this message says that

pex_root

is directly in the

.cache

directory. In CI, I'm using a

pants.ci.toml

file with the following contents:

local_store_dir = ".cache/pants/lmdb_store"
named_caches_dir = ".cache/pants/named_caches"

pants_ignore = [
  ".cache/pants/named_caches",
  ".cache/pants/lmdb_store",
]

But otherwise, I don't think I'm doing anything else different.

I suspect you may be hitting https://github.com/pantsbuild/pex/issues/1252

Ah, if you click through to the linked Pants issue from that Pex issue, indeed that is the exact same Bash error

which now seems rather long 😕

This worked fine with Pants 2.2.0... were there changes with PEX or something else in the meantime that would have changed how this worked?

Yes, Pex added a new venv mode, which reduces nearly all overhead from running a Pex after a one-time cost on the first run by creating a virtualenv for the PEX. It makes Pants much faster with running Pytest, isort, etc

🤔 Is there any Pants configuration to perhaps disable the use of this venv mode?

(just as a temporary thing, in my case)

Update: I was able to tweak the build path for my CI jobs so it's very short now, and the shebang lines work fine. One semi-unfortunate side effect I've discovered, though, is that my CI caching of the Pants cache is now complicated somewhat. On each machine in my build cluster, I can have 1-n agents running and servicing jobs. To keep them from stepping on each other, they each have their own build directory. The caching mechanism currently can only cache things relative to the build directory itself, and not some global location (like the build agent's $HOME directory). Ultimately, this means that my Pants caches will be in the agent-specific build directory; something like

/builds/1/cache

,

/builds/2/cache

, etc. This in turn means that the shebang lines in the cached PEX venvs will also have this agent specificity. It seems I will now need a cache per agent to ensure that the restored caches have PEX venvs that refer to the appropriate path (while experimenting with this, I had some builds that failed because they were using a cache with shebang lines that pointed to a nonexistent path). At the end of the day, it's not a big deal (S3 storage for the caches is cheap, and I don't know that I'm going to have more than a handful of agents on any given box), but it was an interesting realization. I'm not mentioning this to request any kind of change in Pants or anything, just more of an FYI. Thanks again for the assistance on this; I really appreciate it!

I'm not fully following the various CI environment restrictions, but the way to go at 1st blush is to configure https://www.pantsbuild.org/docs/reference-global#section-named-caches-dir to a short global path like

/var/cache/pants

for each agent (this is safe to do, named caches must be concurrency safe) and then if you can't sync a cache from a global location, symlink that directory into an agent-local location.