https://pantsbuild.org/ logo
#general
Title
# general
b

bulky-evening-62934

04/21/2020, 10:59 AM
Hi, I have,
Copy code
[python-setup]
interpreter_constraints = ["CPython>=3.7"]
interpreter_search_paths = ["<PYENV>"]
And I’m wondering what’s the rule of pants to select the interpreter if I have multiple versions in pyenv ? I’m asking this because I have both
3.7.5
and
anaconda3-2019.10
in pyenv, for pants, I hope it could be always the first one got selected, but pants sometimes use the second one (which had some issues with python.h …)
h

hundreds-father-404

04/21/2020, 3:02 PM
Hm, so
interpreter_search_paths
will determine the PATH environment variable that we use with the subprocess, so it should impact Pex’s behavior. I think the issue here is using too imprecise of an environment variable. “<PYENV>” will still allow Pants to use both because it sounds like 3.7.5 is also from pyenv? Instead, try this to find the specific path:
Copy code
$ pyenv which python3.7
/Users/eric/.pyenv/versions/3.7.6/bin/python3.7
then
Copy code
[python-setup]
interpreter_search_paths = ["~/.pyenv/versions/3.7.6/bin"]
I’m not sure how well this would scale if there are other users in your organization. There may be some way to make this more generic. But, let us know if it works and then I can ask others to weigh in
b

bulky-evening-62934

04/21/2020, 3:48 PM
pyenv which python3.7
will point to
~/.pyenv/versions/3.7.5/bin/python3.7
, which is what I want But
"~/.pyenv/versions/3.7.5/bin"
does not scale well, does “$PYENV_ROOT/versions/3.7.5/bin” work ? On the other hand, I have a
.python-version
in the root of my repo which points to
3.7.5
, so I expect pants could always use the python of
pyenv version
Could you tell me where is the code that Pex’s subprocess determine the PATH for python interpreter ?
h

hundreds-father-404

04/21/2020, 3:53 PM
Can you confirm that
~/.pyenv/versions/3.7.5/bin/python3.7
does fix the issue, even though it doesn’t scale well? That will allow us to then find a better solution, like checking if
$PYENV_ROOT/versions/3.7.5/bin
will work GitHub is down, but see
pex/pex_bootstrapper.py
and then
_select_pex_python_interpreter()
b

bulky-evening-62934

04/21/2020, 4:01 PM
~/.pyenv/versions/3.7.5/bin/python3.7
doesn’t work, but absolute path works
😕 1
like
/Users/XXXX/.pyenv/versions/3.7.5/bin
@hundreds-father-404 looking at this function https://github.com/pantsbuild/pants/blob/0d8fdf7c125c824828df3cbff7947af0193153c2/src/python/pants/python/python_setup.py#L228-L252, if I’m using
"<PYENV>"
, it will return the absolute path of all existing pyenv versions as a list
expanded
Then this is used in 2 places, https://github.com/pantsbuild/pants/blob/405b9e3fe34b12bccee5b3e93acbe16efccacd54/src/python/pants/backend/python/interpreter_cache.py#L177 and https://github.com/pantsbuild/pants/blob/0b8fd78923e4b5b211781f53bdc807ed74a7857f/src/python/pants/backend/python/rules/hermetic_pex.py#L53 So the first one is creating the cache and second one is the actual pex process ? why the second one doesn’t use interpreter_constraints as filters ?
h

hundreds-father-404

04/21/2020, 4:38 PM
The first is V1 code that is ignored by what you’re using. The second one is what is relevant. V2 code has a couple places where it influences how Pex chooses interpreters: 1) Where you discovered in hermetic_pex.py, which will influence what PEX sees on the PATH. In V2, subprocesses are almost entirely hermetic by default - they don’t leak things like your environment variables, unless we explicitly set that up. This is why something like
.python-version
will have no impact. 2) Interpreter constraints. See
pex.py
134-138 in Pants 3) Other options like
--python-setup-indexes
or
--not-zip-safe
. See
python_create_binary.py
L47-59 for some of these.
👍 1
b

bulky-evening-62934

04/22/2020, 8:41 AM
Following our discussion, I found when building the binary, the value set in
interpreter_constraints
is ignored, since it’s only used as an option to pex … (I deleted all versions in pyenv, and installed only 3.6.10, even I have constraints like
"CPython>=3.7",
, the binary could be built successfully) So, my current workable solution is having only 3.7.5 installed in pyenv, to make sure the PATH in pex subprocess only have that version presented. I don’t want to use
"<PEXRC>"
or
"<PATH>"
, which is a bit inconvenient … I’m wondering whether we could add a new
special_strings
in
expand_interpreter_search_paths
like
"<PYENV_LOCAL>"
, to let pants use the interpreter in
.python-version
of the
build_root
OR use
os.path.expandvars
before append the path into expanded ? https://github.com/pantsbuild/pants/blob/0d8fdf7c125c824828df3cbff7947af0193153c2/src/python/pants/python/python_setup.py#L244 I guess the second solution is more acceptable ? (and also very straightforward, but I might miss something)
Hi @hundreds-father-404 any thought on my idea ?
h

hundreds-father-404

04/22/2020, 3:32 PM
Hi! Oh sorry, missed this when I woke up. Hm. Oh, wait, there might be a simpler solution. What happens if you set interpreter constraints to
CPython==3.7.5
? (Which, btw, you can use the simpler
==3.7.5
for the same thing) If that doesn’t work, the bigger issue is why interpreter constraints aren’t being used properly, rather than how to change the PATH to force it to use the one you want. -- Also, are you using
platforms
on this
python_binary
?
b

bulky-evening-62934

04/22/2020, 3:33 PM
yes I’m using platform on this python binary, so interpreter constraint will not be passed right ?
h

hundreds-father-404

04/22/2020, 3:37 PM
Yes, that was the change from last week so that we stopped using the
current
platform when building, if you recall your question about why both macOS and linux wheels were being included in your binary. In https://github.com/pantsbuild/pex/issues/957, one of the authors of Pex, John, explains that uses
--interpreter-constraint
signals that you want to be able to locally build things, so it always uses
current
. Which
platform
are you using here? I don’t know Anaconda well, but I suspect
anaconda3-2019.10
works with the platform string you’re specifying, which surprises me.
b

bulky-evening-62934

04/22/2020, 3:39 PM
I’m using
current
and
linux-x86_64-cp-37-cp37m
, the python interpreter is
3.7.5
(not anaconda, I don’t want
anaconda
to be used)
I’m wondering for this line https://github.com/pantsbuild/pants/blob/0d8fdf7c125c824828df3cbff7947af0193153c2/src/python/pants/python/python_setup.py#L244, changing it from
expanded.append(s)
to
expanded.append(os.path.expandvars(s))
could let me specify the
interpreter_search_paths = ["$PYENV_ROOT/versions/3.7.5/bin"]
works
basically the problem is we could only use absolute path in
interpreter_search_paths
, make this change could let us use the path with enviroment variables
h

hundreds-father-404

04/22/2020, 3:43 PM
Hm, I think that would work but need to sanity check it’s safe. @witty-crayon-22786 do you see any issues with the above message to change how we interpret
--python-setup-interpreter-search-paths
? That does make it much more useful.
🙂 1
w

witty-crayon-22786

04/22/2020, 3:44 PM
that sounds reasonable, yea.
h

hundreds-father-404

04/22/2020, 3:45 PM
Awesome. @bulky-evening-62934 would you be interested in putting up a pull request to change this? If so, see https://www.pantsbuild.org/howto_contribute.html for a guide on how to contribute PRs.
b

bulky-evening-62934

04/22/2020, 3:46 PM
Sounds cool, I will do it
👖 1
h

hundreds-father-404

04/22/2020, 3:47 PM
Wonderful. Let us know if you run into any issues getting this all set up. Thanks!
👌 1
b

bulky-evening-62934

04/22/2020, 5:26 PM
@hundreds-father-404 I sent a PR https://github.com/pantsbuild/pants/pull/9613
💯 1
2 Views