https://pantsbuild.org/ logo
Join Slack
Powered by
Is there a way to completely disable pants to see/...
# general
w
Is there a way to completely disable pants to see/try to use xcode’s Python? I’m forced to use MacOS here at work and dont have much experience with it.. I’m trying to migrate to pants v2 and for some reason it’s not using `pyenv`:
Copy code
pex.executor.ExecutableNotFound: caught OSError(2, 'No such file or directory') while trying to execute `[u'/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/bin/python3',
f
I had the same issue I explained there https://github.com/pantsbuild/pants/issues/9760 I went a bit forward in this https://github.com/pantsbuild/pants/pull/10489 But it never worked back then..! So if you happen to find a solution, or if it is solved since 
v2.0.0.dev9
, I would be very interested!
😢 1
The solution I thought would work (but didn't, I don't know why) was to specifiy the following in the 
pants.toml
Copy code
[python-setup]
interpreter_constraints = ["CPython>=xxx"]
interpreter_search_paths = ["<PYENV>"]

[pex]
executable_search_paths =  ["<PATH>"]
w
just increased my hatred towards Apple products haha, just me venting here 😛 miss my linux ❤️ during worktime
😂 1
f
Of course there is the dirty way of tightening the constraints to filter out you mac python, which is what I am currently doing..!
w
would you be kind to share that? haha
h
@hundreds-father-404 you had some ideas here, right?
The MacOS bundled Python is the worst
🍎 1
h
Oof, Etienne, I’m really sorry about that. Realized I didn’t reply to your comment back in August 😞 Will take a look in a little Thales, first try this:
Copy code
[python-setup]
interpreter_search_paths = ["<PYENV>"]

[pex]
executable_search_paths =  ["<PATH>"]
This will tell Pex to only look at Pyenv when choosing an interpreter. Any subprocess you run will set 
PATH
to your original PATH. This means that even if Pex is using the right interpreter, if a subprocess spawned by Pex looks for Python interpreters, it can still choose the bad version. So it might work, but might not depending on your setup
🙏 2
The workaround is to constrain 
interpreter_constraints
to something that avoids the problematic macOS version. For example, 
[">=3.7.3"]
(I think the bad version is 3.7.2 iirc) This is not good though that it’s not very portable. It’s definitely a hacky workaround
w
Copy code
15:13:08.20 [DEBUG] Completed: Searching for `python` on PATH=/Users/lzfnyy/.pyenv/versions/3.6.11/bin:/Users/lzfnyy/.pyenv/versions/3.7.8/bin:/Users/lzfnyy/.pyenv/versions/3.8.1/bin:/Users/lzfnyy/.pyenv/versions/3.8.2/bin
15:13:08.20 [DEBUG] Completed: Searching for `python` on PATH=/Users/lzfnyy/.pyenv/versions/3.6.11/bin:/Users/lzfnyy/.pyenv/versions/3.7.8/bin:/Users/lzfnyy/.pyenv/versions/3.8.1/bin:/Users/lzfnyy/.pyenv/versions/3.8.2/bin
15:13:08.20 [DEBUG] Completed: Find binary path - failed to find python
15:13:09.70 [DEBUG] Completed: Searching for `python3` on PATH=/Users/lzfnyy/.pyenv/versions/3.6.11/bin:/Users/lzfnyy/.pyenv/versions/3.7.8/bin:/Users/lzfnyy/.pyenv/versions/3.8.1/bin:/Users/lzfnyy/.pyenv/versions/3.8.2/bin
15:13:09.70 [DEBUG] Completed: Searching for `python3` on PATH=/Users/lzfnyy/.pyenv/versions/3.6.11/bin:/Users/lzfnyy/.pyenv/versions/3.7.8/bin:/Users/lzfnyy/.pyenv/versions/3.8.1/bin:/Users/lzfnyy/.pyenv/versions/3.8.2/bin
15:13:09.70 [DEBUG] Completed: Find binary path - failed to find python3
15:13:11.43 [DEBUG] Completed: Searching for `python2` on PATH=/Users/lzfnyy/.pyenv/versions/3.6.11/bin:/Users/lzfnyy/.pyenv/versions/3.7.8/bin:/Users/lzfnyy/.pyenv/versions/3.8.1/bin:/Users/lzfnyy/.pyenv/versions/3.8.2/bin
15:13:11.43 [DEBUG] Completed: Searching for `python2` on PATH=/Users/lzfnyy/.pyenv/versions/3.6.11/bin:/Users/lzfnyy/.pyenv/versions/3.7.8/bin:/Users/lzfnyy/.pyenv/versions/3.8.1/bin:/Users/lzfnyy/.pyenv/versions/3.8.2/bin
15:13:11.43 [DEBUG] Completed: Find binary path - failed to find python2
just to clarify, do I have to change that 
"<PYENV>"
to my actual 
~/.pyenv
or do I just keep it like that <PYENV>
h
Keep it 
<PYENV>
Can you please confirm that 
$ pyenv root
does the right thing for you?
w
it returned 
/Users/lzfnyy/.pyenv
to me
👍 1
which is my home folder 🙂
h
Oh wait, it does look like 
<PYENV>
is expanding correclty:
Searching … on
… Users/lzfnyy/.pyenv/versions/3.6.11/bin/Users/lzfnyy/.pyenv/versions/3.7.8/bin/Users/lzfnyy/.pyenv/versions/3.8.1/bin:/Users/lzfnyy/.pyenv/versions/3.8.2/bin So, it sounds like the correct places are being searched, but for some reason the interpreter is not being selected 🤔
What is the error message you’re getting with the above config?
w
well it actually gives me this after trying to find python and before executing the code:
Copy code
15:26:26.28 [WARN] Completed: Find PEX Python - No bootstrap Python executable could be found from the option `interpreter_search_paths` in the `[python-setup]` scope. Will attempt to run PEXes directly.
this is the actual error
this is the actual error
h
Okay, which indeed means it did not find an interpreter. Thinking out loud, my first thought was if your Pyenv wasn’t activated to the correct interpreters, e.g. 
pyenv global
. But I don’t think that’d be relevant, as Pants looks at all the 
.pyenv/versions
folders, which is shown by the above debug message
No such file or directory
Oof. I knew macOS Python was broken, but didn’t realize it’s that broken.
w
dumpster fire
🙃 1
😂 1
Does that mean that even though we fixed the CA Cert thing I’ll never be able to use pants v2 and the wonderful newly redesigned rust engine?
The odd thing is that I’m pretty sure when I tested the CA_Cert it kinda ran the tests when I did 
./pants test
but I cant remember for sure
h
No, definitely not. We only gotta figure out what’s going on here. I’m trying to put together a command to test something In the meantime, you could try try setting 
interpreter_search_paths
to 
["/Users/lzfnyy/.pyenv/versions/3.6.11/bin"]
? I don’t think this will fix it because it looks like the interpreter_search_paths were expanding correctly, and that’s not the issue
w
I’m almost considering purging Xcode from my work laptop 😆
💯 1
h
We’ve also considered ways that we can have Pants automatically never use it. It seems to never work correctly
w
I tried what you said and same behavior.. it says it failed to find python2, python3… and then gives me that error from above
👍 1
h
Update, we tried running the same script that Pants uses to determine a bootstrap interpreter. https://github.com/pantsbuild/pants/blob/ee98ef6fa797e8f2e64059363a47734d9f409b90/src/python/pants/backend/python/util_rules/pex_environment.py#L164-L191 Thales ran this directly on some of the 
~/.pyenv
interpreters, and they all had exit code 0 and printed a fingerprint. This means that Pants should recognize them as valid. Further, as seen above in Thale’s debug logging, Pants is claiming to be searching all the correct folders So I’m a little at a loss for what’s going on now. cc @witty-crayon-22786
w
so, the error above looks like “a failure while finding an interpreter” rather than “a failure to find an interpreter”
ie, the “find interpreter” attempt is dying rather than exiting with “not found”
h
There’s first a failure to find the bootstrap interpreter, which is just a warning. That results in running the Pex via shebang. Which then results in the runtime error
w
Hey just tested updating the 
interpreter_constraints
to 
>=3.8
and that actually made it work so it seems like because I have (or had) the 3.7 under my Xcode it’s trying to use it
h
Thales, when you set interpreter constraints to that, are you not getting the warning anymore about failing to find a bootstrap interpreter?
w
@hundreds-father-404: yes, but do you see the distinction i’m drawing? the error above is the lookup “dying” rather than “giving up”. that process is not 
Falliable
but probably should be.
i’ve got to run for about 30 mins, but if the above makes sense i can try to harden that a bit
w
@hundreds-father-404 the same warnings are happening
Copy code
16:29:02.24 [WARN] Completed: Find PEX Python - No bootstrap Python executable could be found from the option `interpreter_search_paths` in the `[python-setup]` scope. Will attempt to run PEXes directly.
👍 1
h
I don’t think so. We look up in two different contexts: 1) find a bootstrap interpreter, and 2) find the interpreter for specific interpreter constraints, which is what fails My concern is why is the bootstrap interpreter not being found. A followup concern is #2, but #1 is my bigger concern
w
not sure if this helps but this is the excerpt where it seems to try to find an interpreter using the constraints and finds it
Untitled
w
@wooden-thailand-8386: is this pants 2.0.0b0 or b1?
w
b1
👍 1
bleeding edge here
🤘 1
w
@hundreds-father-404: regarding the bootstrap binary search: it’s not actually getting to the point of: https://github.com/pantsbuild/pants/blob/ee98ef6fa797e8f2e64059363a47734d9f409b90/src/python/pants/engine/process.py#L536-L549
or, it’s not finding any to actually test.
so the search script is failing rather than the test script.
h
That’s a really good insight. Is that based on the log messages not including “Test binary foo”?
👍 1
w
hey I think that maybe what I had to do in order to have 1.30 working might be related to this… so I had this similar issue before when using 1.30 and the only way I managed to make 1.30 work on my laptop was to move the 
3.7
folder from inside the 
<http://Xcode.app/|Xcode.app/>…/Python/
to a temp folder
w
@wooden-thailand-8386: what does the following return for you?:
Copy code
PATH=/Users/lzfnyy/.pyenv/versions/3.6.11/bin:/Users/lzfnyy/.pyenv/versions/3.7.8/bin:/Users/lzfnyy/.pyenv/versions/3.8.1/bin:/Users/lzfnyy/.pyenv/versions/3.8.2/bin command which -a python
w
I’m wondering if there’s somewhere in this crappy wonderful OS that have that defined somehow and is still returning that it should exist a 3.7 binary at that folder, and that’s why pex is failing
h
Oof, I refuse to accept that as a valid solution. I mean, yay if it’s a temporary workaround that unblocks you. But we need to do better than that.
w
I’ll never blame on you guys, always on Apple products
@witty-crayon-22786 o.O” what on earth… “which: command not found”
oh hahaha you addded that 
command
before 
which
is that on purpose?
w
that’s fine! how about:
Copy code
PATH=/Users/lzfnyy/.pyenv/versions/3.6.11/bin:/Users/lzfnyy/.pyenv/versions/3.7.8/bin:/Users/lzfnyy/.pyenv/versions/3.8.1/bin:/Users/lzfnyy/.pyenv/versions/3.8.2/bin command -v python
yes, on purpose
w
weird.. if I remove that 
command
I get:
Copy code
/Users/lzfnyy/.pyenv/versions/3.6.11/bin/python
/Users/lzfnyy/.pyenv/versions/3.7.8/bin/python
/Users/lzfnyy/.pyenv/versions/3.8.1/bin/python
/Users/lzfnyy/.pyenv/versions/3.8.2/bin/python
the second one returns only:
Copy code
/Users/lzfnyy/.pyenv/versions/3.6.11/bin/python
w
ok, one sec.
how about
Copy code
command -v which ; echo $?
w
Copy code
which
0
w
ok, thank you… this is good data, although it doesn’t yet point to why that script is not locating anything for you. one more thing in a second.
🙏 1
would you mind putting the following in a file `find-binary.sh`:
Copy code
#!/usr/bin/env bash
set -euox pipefail
if command -v which > /dev/null; then
    command which -a $1
else
    command -v $1
fi
then marking it executable:
Copy code
chmod +x find-binary.sh
then running it like:
Copy code
PATH=/Users/lzfnyy/.pyenv/versions/3.6.11/bin:/Users/lzfnyy/.pyenv/versions/3.7.8/bin:/Users/lzfnyy/.pyenv/versions/3.8.1/bin:/Users/lzfnyy/.pyenv/versions/3.8.2/bin ./find-binary.sh python
…i think that i might already see the issue.
i’m expecting that it’s going to tell you:
Copy code
env: bash: No such file or directory
w
env: bash: No such file or directory
w
filing an issue for both of these.
yep. ok. that’s the issue.
❗ 1
w
\o two bugs in a row?
w
yep… sorry for the trouble. this code got completely rewritten in the last two weeks.
w
haha no worries at all, my wife is QA analyst and she’s giving me two thumbs up lol
😂 1
w
tell her we’re going to have release candidates before 2.0.0, and those should actually become more stable!
(possibly starting next week! unclear.)
thanks a ton for working with us on this.
w
haha thank you guys for putting up with me! Let me know if there’s anything else that I can help with
w
@wooden-thailand-8386: is the 3.8 constraint an ok workaround for now? we should be able to fix both of these fairly quickly i think.
w
Not really, Tensorflow 2.1 being picky about 3.8.. I can wait, there’s other stuff that I have to work on monorepo (tests to be more specific) and I wasnt even planning on updating to v2 this soon, got blocked by another issue today and had “free time” haha
no worries, when you guys fix it I’ll use it 🙂
w
ok, then double/triple thank you for helping to validate.
❤️ 2
i believe i fixed the first half of this in https://github.com/pantsbuild/pants/pull/10858, which should go all the way back to allowing the usecase from #9760 to work
❤️ 1
@fierce-france-72742: sorry for the long delay on this one! we’ll likely cut another beta tomorrow.
❤️ 3
f
Cool, thanks!
w
hey folks! closing the loop: 
2.0.0b2
was released over the weekend that should allow the
Copy code
interpreter_search_path=["<PYENV>"]
solution to work here
w
So just get b2 here and go for it? 🤩
w
assuming you have an 
interpreter_search_path
set that selects just PYENV, yea
w
😞 it didnt work
w
@wooden-thailand-8386: can we start a new thread with the error in #CASMF8SJ1 ?
w
absolutely
w
thanks: sorry for the continued trouble
w
no worries at all
h
cc @enough-analyst-54434 for the original context
e
WOW. Ok, the version of virtualenv pantsbuild/setup uses has a bug where sys.executable of python subprocesses it launches contains the path of the Pants virtualenv interpreter and not of the actual interpreter used: https://bugs.python.org/issue22490#msg283859
🙃 1
So, that breaks an assumption in Pex - that sys.executable is accurate.
👍 1
Actually nothing to do with virtualenv - bad test. This only manifests on OSX and only when using "framework" builds - which is exactly the case for @wooden-thailand-8386 .
😢 1
5 Views
Open in Slack
PreviousNext
Powered by