at some point while i was out on leave or perhaps when i upd

Question

at some point while i was out on leave or perhaps when i updated xcode yesterday Pants own integration tests started picking up a broken Python 3 8 install

witty-crayon-22786 · Answer

```pants engine internals scheduler ExecutionError 1 Exception encountered E Engine traceback in select in pants backend codegen protobuf python rules generate python from protobuf self protocol target address spec in pants backend python util rules pex create venv pex in pants backend python util rules pex build pex mypy protobuf pex in pants engine process fallible to exec result or raise Traceback most recent call last File private var folders bg r10hqp14kjcpv68yzdk5svc0000gn T process executionJZNVWw src python pants engine process py line 262 in fallible to exec result or raise description value pants engine process ProcessExecutionFailure Process Building mypy protobuf pex with 1 requirement mypy protobuf==1 24 failed with exit code 1 stdout E stderr Traceback most recent call last File Users stuhood cache pants named caches pex root venvs 5ebcc58c6baabc17b79b1341a28efd329000cab2 f28b3dbba3c9dae1b4357adde5b079b8b3ca9fac pex line 120 in lt module gt module = importlib import module module name File Applications Xcode app Contents Developer Library Frameworks Python3 framework Versions 3 8 lib python3 8 importlib init py line 127 in import module return bootstrap gcd import name level package level File lt frozen importlib bootstrap gt line 1014 in gcd import File lt frozen importlib bootstrap gt line 991 in find and load File lt frozen importlib bootstrap gt line 961 in find and load unlocked File lt frozen importlib bootstrap gt line 219 in call with frames removed File lt frozen importlib bootstrap gt line 1014 in gcd import File lt frozen importlib bootstrap gt line 991 in find and load File lt frozen importlib bootstrap gt line 961 in find and load unlocked File lt frozen importlib bootstrap gt line 219 in call with frames removed File lt frozen importlib bootstrap gt line 1014 in gcd import File lt frozen importlib bootstrap gt line 991 in find and load File lt frozen importlib bootstrap gt line 961 in find and load unlocked File lt frozen importlib bootstrap gt line 219 in call with frames removed File lt frozen importlib bootstrap gt line 1014 in gcd import File lt frozen importlib bootstrap gt line 991 in find and load File lt frozen importlib bootstrap gt line 973 in find and load unlocked ModuleNotFoundError No module named pip Traceback most recent call last File Users stuhood cache pants named caches pex root unzipped pexes 478cc1fa371ca40aa3e7dafee735ca438d4a243f bootstrap pex pex py line 484 in execute exit value = self wrap coverage self wrap profiling self execute File Users stuhood cache pants named caches pex root unzipped pexes 478cc1fa371ca40aa3e7dafee735ca438d4a243f bootstrap pex pex py line 401 in wrap coverage return runner args File Users stuhood cache pants named caches pex root unzipped pexes 478cc1fa371ca40aa3e7dafee735ca438d4a243f bootstrap pex pex py line 432 in wrap profiling return runner args File Users stuhood cache pants named caches pex root unzipped pexes 478cc1fa371ca40aa3e7dafee735ca438d4a243f bootstrap pex pex py line 540 in execute return self execute entry self pex info entry point File Users stuhood cache pants named caches pex root unzipped pexes 478cc1fa371ca40aa3e7dafee735ca438d4a243f bootstrap pex pex py line 656 in execute entry return self execute pkg resources entry point File Users stuhood cache pants named caches pex root unzipped pexes 478cc1fa371ca40aa3e7dafee735ca438d4a243f bootstrap pex pex py line 688 in execute pkg resources return runner File Users stuhood cache pants named caches pex root unzipped pexes 478cc1fa371ca40aa3e7dafee735ca438d4a243f deps pex 2 1 42 py2 py3 none any whl pex bin pex py line 1145 in main pex builder = build pex options requirements options cache=ENV PEX ROOT File Users stuhood cache pants named caches pex root unzipped pexes 478cc1fa371ca40aa3e7dafee735ca438d4a243f deps pex 2 1 42 py2 py3 none any whl pex bin pex py line 1041 in build pex ignore errors=options ignore errors File Users stuhood cache pants named caches pex root unzipped pexes 478cc1fa371ca40aa3e7dafee735ca438d4a243f deps pex 2 1 42 py2 py3 none any whl pex resolver py line 1034 in resolve multi max parallel jobs=max parallel jobs File Users stuhood cache pants named caches pex root unzipped pexes 478cc1fa371ca40aa3e7dafee735ca438d4a243f deps pex 2 1 42 py2 py3 none any whl pex resolver py line 1134 in download internal dest=dest max parallel jobs=max parallel jobs File Users stuhood cache pants named caches pex root unzipped pexes 478cc1fa371ca40aa3e7dafee735ca438d4a243f deps pex 2 1 42 py2 py3 none any whl pex resolver py line 141 in download distributions max jobs=max parallel jobs File Users stuhood cache pants named caches pex root unzipped pexes 478cc1fa371ca40aa3e7dafee735ca438d4a243f deps pex 2 1 42 py2 py3 none any whl pex jobs py line 431 in execute parallel yield spawn result spawned job await result File Users stuhood cache pants named caches pex root unzipped pexes 478cc1fa371ca40aa3e7dafee735ca438d4a243f deps pex 2 1 42 py2 py3 none any whl pex jobs py line 185 in await result job wait File Users stuhood cache pants named caches pex root unzipped pexes 478cc1fa371ca40aa3e7dafee735ca438d4a243f deps pex 2 1 42 py2 py3 none any whl pex jobs py line 74 in wait self check returncode File Users stuhood cache pants named caches pex root unzipped pexes 478cc1fa371ca40aa3e7dafee735ca438d4a243f deps pex 2 1 42 py2 py3 none any whl pex pip py line 502 in check returncode with open self log r as fp FileNotFoundError Errno 2 No such file or directory private tmp process execution2dGCfW tmp tmpa6kwwtw pip log ```

witty-crayon-22786 · Answer

so this is maybe a useful opportunity to figure out how to fix for this install

enough-analyst-54434 · Answer

Hw do you know the python 3 8 install is broken That s not obvious to me from the backtrace

witty-crayon-22786 · Answer

i do not i suppose hence but the `ModuleNotFoundError No module named pip ` in a case where we expect it to be vendored into pex felt very unusual

enough-analyst-54434 · Answer

Yeah it seems suspicious but digging is definitely needed before blaming the interpreter

witty-crayon-22786 · Answer

i m going to pursue skipping this one but if it s worth opening a ticket for i m happy to work on gathering more details

enough-analyst-54434 · Answer

If you re willing to dig I encourage it Every single opportunity we skip digging and solving seems to turn into a user reported problem eventually

witty-crayon-22786 · Answer

is there a default set of please attach the output of these commands that would be useful to attach to a PEX ticket to start with

enough-analyst-54434 · Answer

Well the advantage here is I have you and not a typical user I m hoping you can use everything you know to dig into the relevant details

enough-analyst-54434 · Answer

Like starting from the top and not knowing anything about anything I t would be awesome to add prints to line 120 of Users stuhood cache pants named caches pex root venvs 5ebcc58c6baabc17b79b1341a28efd329000cab2 f28b3dbba3c9dae1b4357adde5b079b8b3ca9fac pex

enough-analyst-54434 · Answer

Or just cracking open that file snad staring at it Its an venv excutable so must have a shebang What python does it think it should work with Is it using a wildly different one

witty-crayon-22786 · Answer

``` Users stuhood cache pants named caches pex root venvs 5ebcc58c6baabc17b79b1341a28efd329000cab2 f28b3dbba3c9dae1b4357adde5b079b8b3ca9fac pex``` has a shebang for ``` Users stuhood cache pants named caches pex root venvs short 629c52d2 bin python3 gt Applications Xcode app Contents Developer usr bin python3```

enough-analyst-54434 · Answer

The issue with all these macOS reports is I don t have one I really need one or more macOS users to be able to debug these things Otherwise its just massively painful It takes a huge amount of back and forth and groping in th dark

witty-crayon-22786 · Answer

yea we need to foster some more contributors who use macOS

enough-analyst-54434 · Answer

Ok and that s probably a python3 that changed when you updated XCode

witty-crayon-22786 · Answer

yes i ve never seen it before so it must have been included in more recent versions

enough-analyst-54434 · Answer

Um that s not what I mean I m positing that path existed before and after your XCode updaTE BUT WITH DIFFERENT CONTENTS BEFORE AND AFTER

enough-analyst-54434 · Answer

Oops on the stray caps lock

witty-crayon-22786 · Answer

i could believe that ` Applications Xcode app Contents Developer usr bin python3` existed yea it is also a symlink sorry should have walked that sec

witty-crayon-22786 · Answer

symlink chain ``` Users stuhood cache pants named caches pex root venvs short 629c52d2 bin python3 Applications Xcode app Contents Developer usr bin python3 Applications Xcode app Contents Developer Library Frameworks Python3 framework Versions 3 8 bin python3 Applications Xcode app Contents Developer Library Frameworks Python3 framework Versions 3 8 bin python3 8```

enough-analyst-54434 · Answer

To highlight though this is what I mean I ve gone through hundreds of these sesssion at this point and solved lots of macOS specific bugs but its super painful I think someone like you has the skills to take this home

enough-analyst-54434 · Answer

Ok yeah so I was brainstorming but wondering is a new python3 at that same path is what is going on Totally handwavy

enough-analyst-54434 · Answer

So next step would be does the venv look like it contains pip

witty-crayon-22786 · Answer

it does not look like it contains pip ```$ ll Users stuhood cache pants named caches pex root venvs 5ebcc58c6baabc17b79b1341a28efd329000cab2 f28b3dbba3c9dae1b4357adde5b079b8b3ca9fac bin total 24 rw r r 1 stuhood staff 2451 Jun 14 10 21 activate rw r r 1 stuhood staff 1507 Jun 14 10 21 activate csh rw r r 1 stuhood staff 2659 Jun 14 10 21 activate fish lrwxrwx 1 stuhood staff 7 Jun 14 10 21 python gt python3 lrwxrwx 1 stuhood staff 58 Jun 14 10 21 python3 gt Applications Xcode app Contents Developer usr bin python3```

enough-analyst-54434 · Answer

nono

enough-analyst-54434 · Answer

lib python3 8 site packages

enough-analyst-54434 · Answer

Not the script

witty-crayon-22786 · Answer

it contains neither the script nor the module afaict

witty-crayon-22786 · Answer

``` Users stuhood cache pants named caches pex root venvs 5ebcc58c6baabc17b79b1341a28efd329000cab2 f28b3dbba3c9dae1b4357adde5b079b8b3ca9fac bin python3 Python 3 8 2 default Apr 8 2021 23 19 18 Clang 12 0 5 clang 1205 0 22 9 on darwin Type help copyright credits or license for more information gt gt gt import pip Traceback most recent call last File lt stdin gt line 1 in lt module gt ModuleNotFoundError No module named pip ```

enough-analyst-54434 · Answer

Ok so now your step is to see where that import pip is coming from

enough-analyst-54434 · Answer

What is this venv trying to do when pantas executes it

enough-analyst-54434 · Answer

Or is pants even executing it Is Pants executing pex and pex then executing it

enough-analyst-54434 · Answer

The venv is in the pex root so it could be either

enough-analyst-54434 · Answer

What is in the venv site packages

witty-crayon-22786 · Answer

from the stacktrace it looks like PEX is attempting to launch a PEX containing `pip` and that is failing i e we did not successfully build a venv containing `pip` should i go backwards to see whether the pex file which was extracted into a venv actually contained pip

enough-analyst-54434 · Answer

How about 1st listing whats in the venv site packages as the more basic step That will reveal what the venv is actuall of

enough-analyst-54434 · Answer

Its a good sanity check

witty-crayon-22786 · Answer

oh interesting there is no python3 8 site packages there is only a python3 7 site packages

witty-crayon-22786 · Answer

so the venv was built for a different interpreter and then the symlink changed

enough-analyst-54434 · Answer

Aha my original gut Yes

enough-analyst-54434 · Answer

I think thats enough for me to reason about whther this is a straight up pex bug or a more subtle bastard macOS bug for Pex to work around

enough-analyst-54434 · Answer

Thank you

witty-crayon-22786 · Answer

ok so a very unhelpful workaround would be for me to wipe the `pex root` should i file an issue with more info first

hundreds-father-404 · Answer

or rename it < witty crayon 22786>

witty-crayon-22786 · Answer

yea doing that for now

enough-analyst-54434 · Answer

I m trying a repro 1st This may still be down to macOS shenanigans but let s see If I can repro I ll file an issue If not I may grab details from here

enough-analyst-54434 · Answer

I can t get a repro by redirecting a symlink to simulate your upgrade I m guessing the upgrade went from shim binary gt symlink I ll try to simulate that If XCode were open source I could check to see if this is what happened of course

enough-analyst-54434 · Answer

Yeah can t get that to repro either This is macOS shenanigans afaict We have everything we need to get a repro though It will just take some fortitude 1 Determine the XCode from gt to update 2 Setup up a macOS vm on a macOS machine using the from XCode and run Pants 3 Inspect the heck out of the python 3 system interpreter being picked here 4 Upgrade to the to XCode and reproduce the issue 5 Start mucking with Pex in a repl back on the from XCode vm snapshot to see how it identifies that interpreter vs the to This is all illegal for me to do but maybe one of you with macOS has the fortitude I d love to solve it and fully understand it as with all bugs like this

witty-crayon-22786 · Answer

looks like you can install multiple xcodes on one box and then toggle between them using `xcode select` hold onto your butts

witty-crayon-22786 · Answer

going to be focused on a talk so snoozing this until next week

witty-crayon-22786 · Answer

opened <https github com pantsbuild pex issues 1386> about this it s still snoozed though = soon