https://pantsbuild.org/ logo
Join Slack
Powered by
at some point while i was out on leave (or perhaps...
# development
w
at some point while i was out on leave (or perhaps when i updated xcode yesterday), Pants’ own integration tests started picking up a broken (?) Python 3.8 install
Copy code
pants.engine.internals.scheduler.ExecutionError: 1 Exception encountered:
E
Engine traceback:
  in select
  in pants.backend.codegen.protobuf.python.rules.generate_python_from_protobuf ({self.protocol_target.address.spec})
  in pants.backend.python.util_rules.pex.create_venv_pex
  in pants.backend.python.util_rules.pex.build_pex (mypy_protobuf.pex)
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/private/var/folders/bg/_r10hqp14kjcpv68yzdk5svc0000gn/T/process-executionJZNVWw/src/python/pants/engine/process.py", line 262, in fallible_to_exec_result_or_raise
    description.value,
pants.engine.process.ProcessExecutionFailure: Process 'Building mypy_protobuf.pex with 1 requirement: mypy-protobuf==1.24' failed with exit code 1.
stdout:
E
stderr:
Traceback (most recent call last):
  File "/Users/stuhood/.cache/pants/named_caches/pex_root/venvs/5ebcc58c6baabc17b79b1341a28efd329000cab2/f28b3dbba3c9dae1b4357adde5b079b8b3ca9fac/pex", line 120, in <module>
    module = importlib.import_module(module_name)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'pip'
Traceback (most recent call last):
  File "/Users/stuhood/.cache/pants/named_caches/pex_root/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.bootstrap/pex/pex.py", line 484, in execute
    exit_value = self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/Users/stuhood/.cache/pants/named_caches/pex_root/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.bootstrap/pex/pex.py", line 401, in _wrap_coverage
    return runner(*args)
  File "/Users/stuhood/.cache/pants/named_caches/pex_root/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.bootstrap/pex/pex.py", line 432, in _wrap_profiling
    return runner(*args)
  File "/Users/stuhood/.cache/pants/named_caches/pex_root/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.bootstrap/pex/pex.py", line 540, in _execute
    return self.execute_entry(self._pex_info.entry_point)
  File "/Users/stuhood/.cache/pants/named_caches/pex_root/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.bootstrap/pex/pex.py", line 656, in execute_entry
    return self.execute_pkg_resources(entry_point)
  File "/Users/stuhood/.cache/pants/named_caches/pex_root/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.bootstrap/pex/pex.py", line 688, in execute_pkg_resources
    return runner()
  File "/Users/stuhood/.cache/pants/named_caches/pex_root/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.deps/pex-2.1.42-py2.py3-none-any.whl/pex/bin/pex.py", line 1145, in main
    pex_builder = build_pex(options.requirements, options, cache=ENV.PEX_ROOT)
  File "/Users/stuhood/.cache/pants/named_caches/pex_root/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.deps/pex-2.1.42-py2.py3-none-any.whl/pex/bin/pex.py", line 1041, in build_pex
    ignore_errors=options.ignore_errors,
  File "/Users/stuhood/.cache/pants/named_caches/pex_root/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.deps/pex-2.1.42-py2.py3-none-any.whl/pex/resolver.py", line 1034, in resolve_multi
    max_parallel_jobs=max_parallel_jobs,
  File "/Users/stuhood/.cache/pants/named_caches/pex_root/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.deps/pex-2.1.42-py2.py3-none-any.whl/pex/resolver.py", line 1134, in _download_internal
    dest=dest, max_parallel_jobs=max_parallel_jobs
  File "/Users/stuhood/.cache/pants/named_caches/pex_root/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.deps/pex-2.1.42-py2.py3-none-any.whl/pex/resolver.py", line 141, in download_distributions
    max_jobs=max_parallel_jobs,
  File "/Users/stuhood/.cache/pants/named_caches/pex_root/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.deps/pex-2.1.42-py2.py3-none-any.whl/pex/jobs.py", line 431, in execute_parallel
    yield spawn_result.spawned_job.await_result()
  File "/Users/stuhood/.cache/pants/named_caches/pex_root/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.deps/pex-2.1.42-py2.py3-none-any.whl/pex/jobs.py", line 185, in await_result
    job.wait()
  File "/Users/stuhood/.cache/pants/named_caches/pex_root/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.deps/pex-2.1.42-py2.py3-none-any.whl/pex/jobs.py", line 74, in wait
    self._check_returncode()
  File "/Users/stuhood/.cache/pants/named_caches/pex_root/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.deps/pex-2.1.42-py2.py3-none-any.whl/pex/pip.py", line 502, in _check_returncode
    with open(self._log, "r") as fp:
FileNotFoundError: [Errno 2] No such file or directory: '/private/tmp/process-execution2dGCfW/.tmp/tmpa6kwwtw_/pip.log'
so… this is maybe a useful opportunity to figure out how to fix for this install.
e
Hw do you know the python 3.8 install is broken? That's not obvious to me from the backtrace.
w
i do not i suppose… hence “(?)“. but the 
ModuleNotFoundError: No module named 'pip'
in a case where we expect it to be vendored into pex felt very unusual.
e
Yeah - it seems suspicious but digging is definitely needed before blaming the interpreter.
w
i’m going to pursue skipping this one, but if it’s worth opening a ticket for i’m happy to work on gathering more details.
e
If you're willing to dig, I encourage it. Every single opportunity we skip digging and solving seems to turn into a user-reported problem eventually.
➕ 1
w
is there a default set of “please attach the output of these commands” that would be useful to attach to a PEX ticket to start with?
e
Well, the advantage here is I have you and not a typical user. I'm hoping you can use everything you know to dig into the relevant details.
Like, starting from the top and not knowing anything about anything. I t would be awesome to add prints to line 120 of "/Users/stuhood/.cache/pants/named_caches/pex_root/venvs/5ebcc58c6baabc17b79b1341a28efd329000cab2/f28b3dbba3c9dae1b4357adde5b079b8b3ca9fac/pex"
👍 1
Or just cracking open that file snad staring at it. Its an venv excutable so must have a shebang. What python does it think it should work with? Is it using a wildly different one?
w
Copy code
/Users/stuhood/.cache/pants/named_caches/pex_root/venvs/5ebcc58c6baabc17b79b1341a28efd329000cab2/f28b3dbba3c9dae1b4357adde5b079b8b3ca9fac/pex
has a shebang for:
Copy code
/Users/stuhood/.cache/pants/named_caches/pex_root/venvs/short/629c52d2/bin/python3 -> /Applications/Xcode.app/Contents/Developer/usr/bin/python3
e
The issue with all these macOS reports is I don't have one. I really need one or more macOS users to be able to debug these things. Otherwise its - just massively painful. It takes a huge amount of back and forth and groping in th dark.
w
yea. we need to foster some more contributors who use macOS.
e
Ok - and that's probably a python3 that changed when you updated XCode?
w
yes… i’ve never seen it before, so it must have been included in more recent versions.
e
Um, that's not what I mean. I'm positing that path existed before and after your XCode updaTE BUT WITH DIFFERENT CONTENTS BEFORE AND AFTER
Oops on the stray caps-lock!
w
i could believe that 
/Applications/Xcode.app/Contents/Developer/usr/bin/python3
existed, yea… it is also a symlink. sorry, should have walked that. sec.
symlink chain:
Copy code
/Users/stuhood/.cache/pants/named_caches/pex_root/venvs/short/629c52d2/bin/python3
/Applications/Xcode.app/Contents/Developer/usr/bin/python3
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/bin/python3
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/bin/python3.8
e
To highlight, though - this is what I mean. I've gone through hundreds of these sesssion at this point and solved lots of macOS specific bugs, but its super painful. I think someone like you has the skills to take this home.
Ok, yeah - so ... I was brainstorming, but wondering is a new python3 at that same path is what is going on. Totally handwavy.
So next step would be - does the venv look like it contains pip?
w
it does not look like it contains pip:
Copy code
$ ll /Users/stuhood/.cache/pants/named_caches/pex_root/venvs/5ebcc58c6baabc17b79b1341a28efd329000cab2/f28b3dbba3c9dae1b4357adde5b079b8b3ca9fac/bin/
total 24
-rw-r--r--  1 stuhood  staff  2451 Jun 14 10:21 activate
-rw-r--r--  1 stuhood  staff  1507 Jun 14 10:21 activate.csh
-rw-r--r--  1 stuhood  staff  2659 Jun 14 10:21 activate.fish
lrwxrwx---  1 stuhood  staff     7 Jun 14 10:21 python -> python3
lrwxrwx---  1 stuhood  staff    58 Jun 14 10:21 python3 -> /Applications/Xcode.app/Contents/Developer/usr/bin/python3
e
nono
lib/python3.8/site-packages.
Not the script
w
it contains neither the script nor the module afaict.
Copy code
/Users/stuhood/.cache/pants/named_caches/pex_root/venvs/5ebcc58c6baabc17b79b1341a28efd329000cab2/f28b3dbba3c9dae1b4357adde5b079b8b3ca9fac/bin/python3
Python 3.8.2 (default, Apr  8 2021, 23:19:18)
[Clang 12.0.5 (clang-1205.0.22.9)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pip
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'pip'
e
Ok, so now your step is to see where that import pip is coming from
What is this venv trying to do when pantas executes it?
Or is pants even executing it? Is Pants executing pex and pex then executing it?
The venv is in the pex_root, so it could be either.
What is in the venv site-packages?
w
from the stacktrace, it looks like PEX is attempting to launch a PEX containing 
pip
, and that is failing. i.e., we did not successfully build a venv containing 
pip
. should i go backwards to see whether the pex file which was extracted into a venv actually contained pip?
e
How about 1st listing whats in the venv site-packages as the more basic step. That will reveal what the venv is actuall of.
Its a good sanity check.
w
…oh, interesting. there is no python3.8 site packages. there is only a python3.7 site-packages.
so the venv was built for a different interpreter, and then the symlink changed…?
e
Aha - my original gut. Yes.
I think thats enough for me to reason about whther this is a straight up pex bug or a more subtle bastard macOS bug for Pex to work around.
Thank you.
w
ok. so a very unhelpful workaround would be for me to wipe the `pex_root`… should i file an issue with more info first?
h
or rename it @witty-crayon-22786
w
yea, doing that for now.
e
I'm trying a repro 1st. This may still be down to macOS shenanigans, but let's see. If I can repro, I'll file an issue. If not I may grab details from here.
🤞 1
I can't get a repro by redirecting a symlink to simulate your upgrade. I'm guessing the upgrade went from shim binary -> symlink. I'll try to simulate that. If XCode were open source I could check to see if this is what happened of course!
Yeah - can't get that to repro either. This is macOS shenanigans afaict. We have everything we need to get a repro though. It will just take some fortitude: 1. Determine the XCode from -> to update. 2. Setup up a macOS vm on a macOS machine using the "from" XCode and run Pants 3. Inspect the heck out of the python 3 system interpreter being picked here. 4. Upgrade to the "to" XCode and reproduce the issue. 5. Start mucking with Pex in a repl back on the "from" XCode vm snapshot to see how it identifies that interpreter vs the "to". This is all illegal for me to do, but maybe one of you with macOS has the fortitude. I'd love to solve it and fully understand it as with all bugs like this.
w
looks like you can install multiple xcodes on one box, and then toggle between them using 
xcode-select
. hold onto your butts.
going to be focused on a talk, so snoozing this until next week.
opened https://github.com/pantsbuild/pex/issues/1386 about this… it’s still snoozed though =( … soon!
👍 1
Open in Slack
PreviousNext
Powered by