https://pantsbuild.org/ logo
Join Slack
Powered by
What would cause an error like the one in the thre...
# general
h
What would cause an error like the one in the thread? It's happening in our CI system. Haven't been successful reproducing it locally. Clearing out 
~/.cache/pants
has not seemed to be helpful.
Copy code
16:57:48.32 [INFO] Scheduler initialized.
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/buildbot/.pex/unzipped_pexes/2aac9312154c299fac910874b778e800dca4c62a/__main__.py", line 102, in <module>
    from pex.pex_bootstrapper import bootstrap_pex
ModuleNotFoundError: No module named 'pex'
16:57:51.47 [ERROR] 1 Exception encountered:
Engine traceback:
  in select
  in pants.core.goals.export.export
Traceback (most recent call last):
  File "/home/buildbot/.cache/pants/setup/bootstrap-Linux-x86_64/2.13.0_py38/lib/python3.8/site-packages/pants/engine/internals/selectors.py", line 653, in native_engine_generator_send
    res = func.send(arg)
  File "/home/buildbot/.cache/pants/setup/bootstrap-Linux-x86_64/2.13.0_py38/lib/python3.8/site-packages/pants/core/goals/export.py", line 142, in export
    raise ExportError(f"Failed to write {result.description} to {result_dir}")
pants.core.goals.export.ExportError: Failed to write virtualenv for the resolve 'python-default' (using Python 3.8.10) to dist/export/python/virtualenvs/python-default
Current thought is that we have multiple workers calling 
./pants export
at different times and messing with the state of the cache.
Would it be recommended to clear out 
~/.pex
as well. I don't remember seeing any notes in the Pants docs about that. Just some notes about monitoring the size of 
~/.cache/pants
.
We're also seeing multiple 
pantsd
processes on the troubled machine
e
@high-yak-85899 can you get onto the buildbot node (or script up a build step) that does recursive listing (maybe just a 
find /home/buildbot/.pex/
)? I'm not sure why that backtrace would be happening after a pantsd bootstrap since - fwict - all Pex CLI use in Pants sets up a 
PEX_ROOT
of 
/home/buildbot/.cache/pants/named_caches/pex_root
, but I'll check the export goal ruleset since that's implicated.
h
We had to clear out the container just now to keep our pipeline going since every build was failing. As an experiment, I implemented a custom file lock around where we call 
./pants export
to protect against any concurrency.
e
OK. Pex does the same for creation of that 
/home/buildbot/.pex/unzipped_pexes/2aac9312154c299fac910874b778e800dca4c62a
direcory (and all other PEX_ROOT directories. It uses: + https://github.com/pantsbuild/pex/blob/bc29962d1b05d514c9098e1f5077734463bb69b3/pex/layout.py#L98-L109 + https://github.com/pantsbuild/pex/blob/bc29962d1b05d514c9098e1f5077734463bb69b3/pex/common.py#L435-L471
As far as the 
~/.pex
leak, it's right here: https://github.com/pantsbuild/pants/blob/2551b3382f9135cdd8ab3c96225def77ff5c2f20/src/python/pants/backend/python/goals/export.py#L178-L195 I'll file an issue - that PEX_ROOT non-specification is an unintended leak / aka Pants using 
/home/buildbot/.pex
is a bug.
h
Great find! Thanks for the quick triage.
e
That find is just a tidy, it has nothing to do with the missing file. That's still an issue here to debug.
In case it wasn't clear.
👍 1
And this is probably of interest to @happy-kitchen-89482 who just recently took a look at export perf which this affects for cold `~/.pex`: https://github.com/pantsbuild/pants/issues/17176
h
I'm also seeing this same kind of 
ModuleNotFoundError: No module named 'pex'
in other places completely separate from the export goal. I'm wondering if this is a systematic issue beyond just 
export
.
e
It's almost certainly another instance of this: https://github.com/pantsbuild/pants/issues/16778 That's unresolved with no one currently working the problem IIUC. I was the last to investigate and after a few man-days of effort have come up with nothing so far. This is an embarrassing and frustrating problem to be sure.
h
Ah, that's unfortunate. Appreciate your efforts on it!
Open in Slack
PreviousNext
Powered by