https://pantsbuild.org/ logo
#general
Title
# general
h

high-yak-85899

10/10/2022, 5:04 PM
What would cause an error like the one in the thread? It's happening in our CI system. Haven't been successful reproducing it locally. Clearing out
~/.cache/pants
has not seemed to be helpful.
Copy code
16:57:48.32 [INFO] Scheduler initialized.
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/buildbot/.pex/unzipped_pexes/2aac9312154c299fac910874b778e800dca4c62a/__main__.py", line 102, in <module>
    from pex.pex_bootstrapper import bootstrap_pex
ModuleNotFoundError: No module named 'pex'
16:57:51.47 [ERROR] 1 Exception encountered:
Engine traceback:
  in select
  in pants.core.goals.export.export
Traceback (most recent call last):
  File "/home/buildbot/.cache/pants/setup/bootstrap-Linux-x86_64/2.13.0_py38/lib/python3.8/site-packages/pants/engine/internals/selectors.py", line 653, in native_engine_generator_send
    res = func.send(arg)
  File "/home/buildbot/.cache/pants/setup/bootstrap-Linux-x86_64/2.13.0_py38/lib/python3.8/site-packages/pants/core/goals/export.py", line 142, in export
    raise ExportError(f"Failed to write {result.description} to {result_dir}")
pants.core.goals.export.ExportError: Failed to write virtualenv for the resolve 'python-default' (using Python 3.8.10) to dist/export/python/virtualenvs/python-default
Current thought is that we have multiple workers calling
./pants export
at different times and messing with the state of the cache.
Would it be recommended to clear out
~/.pex
as well. I don't remember seeing any notes in the Pants docs about that. Just some notes about monitoring the size of
~/.cache/pants
.
We're also seeing multiple
pantsd
processes on the troubled machine
e

enough-analyst-54434

10/10/2022, 5:45 PM
@high-yak-85899 can you get onto the buildbot node (or script up a build step) that does recursive listing (maybe just a
find /home/buildbot/.pex/
)? I'm not sure why that backtrace would be happening after a pantsd bootstrap since - fwict - all Pex CLI use in Pants sets up a
PEX_ROOT
of
/home/buildbot/.cache/pants/named_caches/pex_root
, but I'll check the export goal ruleset since that's implicated.
h

high-yak-85899

10/10/2022, 5:49 PM
We had to clear out the container just now to keep our pipeline going since every build was failing. As an experiment, I implemented a custom file lock around where we call
./pants export
to protect against any concurrency.
e

enough-analyst-54434

10/10/2022, 5:53 PM
OK. Pex does the same for creation of that
/home/buildbot/.pex/unzipped_pexes/2aac9312154c299fac910874b778e800dca4c62a
direcory (and all other PEX_ROOT directories. It uses: + https://github.com/pantsbuild/pex/blob/bc29962d1b05d514c9098e1f5077734463bb69b3/pex/layout.py#L98-L109 + https://github.com/pantsbuild/pex/blob/bc29962d1b05d514c9098e1f5077734463bb69b3/pex/common.py#L435-L471
As far as the
~/.pex
leak, it's right here: https://github.com/pantsbuild/pants/blob/2551b3382f9135cdd8ab3c96225def77ff5c2f20/src/python/pants/backend/python/goals/export.py#L178-L195 I'll file an issue - that PEX_ROOT non-specification is an unintended leak / aka Pants using
/home/buildbot/.pex
is a bug.
h

high-yak-85899

10/10/2022, 5:56 PM
Great find! Thanks for the quick triage.
e

enough-analyst-54434

10/10/2022, 5:57 PM
That find is just a tidy, it has nothing to do with the missing file. That's still an issue here to debug.
In case it wasn't clear.
👍 1
And this is probably of interest to @happy-kitchen-89482 who just recently took a look at export perf which this affects for cold `~/.pex`: https://github.com/pantsbuild/pants/issues/17176
h

high-yak-85899

10/11/2022, 4:33 PM
I'm also seeing this same kind of
ModuleNotFoundError: No module named 'pex'
in other places completely separate from the export goal. I'm wondering if this is a systematic issue beyond just
export
.
e

enough-analyst-54434

10/11/2022, 5:28 PM
It's almost certainly another instance of this: https://github.com/pantsbuild/pants/issues/16778 That's unresolved with no one currently working the problem IIUC. I was the last to investigate and after a few man-days of effort have come up with nothing so far. This is an embarrassing and frustrating problem to be sure.
h

high-yak-85899

10/11/2022, 5:39 PM
Ah, that's unfortunate. Appreciate your efforts on it!