high-yak-85899
10/13/2022, 2:36 PM10:42:12.54 [INFO] Initialization options changed: reinitializing scheduler...
10:42:13.10 [INFO] Scheduler initialized.
Traceback (most recent call last):
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/buildbot/.pex/unzipped_pexes/7907cee4a80399696d31dbee748702a492b91a80/__main__.py", line 102, in <module>
from pex.pex_bootstrapper import bootstrap_pex
ModuleNotFoundError: No module named 'pex'
high-yak-85899
10/13/2022, 2:38 PM./pants run <some cli tool> -- <its necessary arguments>
high-yak-85899
10/13/2022, 2:40 PM.pex
folder so I'm hesitant to just start throwing things like --no-local-cache
at it.happy-kitchen-89482
10/13/2022, 2:49 PMhappy-kitchen-89482
10/13/2022, 2:50 PMhappy-kitchen-89482
10/13/2022, 2:50 PMexport
, sounds likehappy-kitchen-89482
10/13/2022, 2:51 PMhappy-kitchen-89482
10/13/2022, 2:54 PMhigh-yak-85899
10/13/2022, 2:55 PM.pex
directory and retried the goal. But doing that everywhere is going to be tedious.high-yak-85899
10/13/2022, 2:55 PM.pex
directory they have.happy-kitchen-89482
10/13/2022, 2:57 PMwitty-crayon-22786
10/13/2022, 3:55 PM.pex
directory is not the one that pants manageswitty-crayon-22786
10/13/2022, 3:56 PM~/.cache/pants/named_caches/pex_root
)witty-crayon-22786
10/13/2022, 3:56 PMwitty-crayon-22786
10/13/2022, 3:56 PM./pants run $something
witty-crayon-22786
10/13/2022, 3:57 PM~/.pex
directory? and if you ./pants package $something
and then run the pex repeatedly, is it reproducible?high-yak-85899
10/13/2022, 4:31 PMexport
goal, we manually wipe ~/.pex
and then export
succeeds. So there's probably some buried concurrency issue where one ./pants
call is working while this ./pants run
gets invoked.high-yak-85899
10/13/2022, 4:44 PMpantsd
processes when inspecting the machineshigh-yak-85899
10/13/2022, 4:54 PM~/.pex
but clearly still using it, there's no way to reliably prevent collisions if we run multiple jobs on the same machine with different clones of our repository.witty-crayon-22786
10/13/2022, 5:20 PMTo mitigate this happening on ourhow are you manually wiping the directory? are you doing so in a concurrency safe way?goal, we manually wipeexport
and then~/.pex
succeeds.export
witty-crayon-22786
10/13/2022, 5:20 PMwitty-crayon-22786
10/13/2022, 5:23 PMrun
is implemented as (almost) “build a PEX and then run it un-sandboxed”… when you run a PEX outside of Pants’ sandbox, it does not use pants’ config (and shouldn’t)high-yak-85899
10/13/2022, 5:32 PM./pants run
while a separate worker is bootstrapping. It's not clear when this directory becomes invalidated: just that it does.high-yak-85899
10/13/2022, 5:34 PMpackage
before we run
our steps? Or is there some kind of legacy action we can invoke? I'm worried about chasing down all the places we call run
and finding the appropriate dist
path to invoke.witty-crayon-22786
10/13/2022, 5:34 PM.pex
directory while other processes are running ./pants run
? the ./pants
/ pex
processes won’t know anything about the lockwitty-crayon-22786
10/13/2022, 5:35 PMAs a workaround, and maybe this is what you meant above, would you suggest running athat won’t fix the problem, per-se: just allow you to see whether it is reproducible (because the resulting PEX file fails for every attempt) or just a race condition.before wepackage
our steps?run
high-yak-85899
10/13/2022, 5:38 PM.pex
• try again
• release lock
I'm sure we can make it happen, but I wouldn't consider it reliable/easy to implement.witty-crayon-22786
10/13/2022, 5:42 PM.pex
sounds problematic in general. it’s just plain hard in the presence of concurrency
so it seems like backing up and fixing export
so that you don’t need to wipe .pex
would be best. is there an issue filed for the export
issue specifically? https://github.com/pantsbuild/pants/issues/16778#issuecomment-1276294801 is probably an unrelated issuehigh-yak-85899
10/13/2022, 5:43 PMrun
is doing the same thing. This happened before and we only recently implemented this patch to try to remedy the situation.high-yak-85899
10/13/2022, 6:29 PM