Hi, I have,

[python-setup]
interpreter_constraints = ["CPython>=3.7"]
interpreter_search_paths = ["<PYENV>"]

And I’m wondering what’s the rule of pants to select the interpreter if I have multiple versions in pyenv ? I’m asking this because I have both

3.7.5

and

anaconda3-2019.10

in pyenv, for pants, I hope it could be always the first one got selected, but pants sometimes use the second one (which had some issues with python.h …)

Hm, so

interpreter_search_paths

will determine the PATH environment variable that we use with the subprocess, so it should impact Pex’s behavior. I think the issue here is using too imprecise of an environment variable. “<PYENV>” will still allow Pants to use both because it sounds like 3.7.5 is also from pyenv? Instead, try this to find the specific path:

$ pyenv which python3.7
/Users/eric/.pyenv/versions/3.7.6/bin/python3.7

then

[python-setup]
interpreter_search_paths = ["~/.pyenv/versions/3.7.6/bin"]

I’m not sure how well this would scale if there are other users in your organization. There may be some way to make this more generic. But, let us know if it works and then I can ask others to weigh in

pyenv which python3.7

will point to

~/.pyenv/versions/3.7.5/bin/python3.7

, which is what I want But

"~/.pyenv/versions/3.7.5/bin"

does not scale well, does “$PYENV_ROOT/versions/3.7.5/bin” work ? On the other hand, I have a

.python-version

in the root of my repo which points to

3.7.5

, so I expect pants could always use the python of

pyenv version

Could you tell me where is the code that Pex’s subprocess determine the PATH for python interpreter ?

Can you confirm that

~/.pyenv/versions/3.7.5/bin/python3.7

does fix the issue, even though it doesn’t scale well? That will allow us to then find a better solution, like checking if

$PYENV_ROOT/versions/3.7.5/bin

will work GitHub is down, but see

pex/pex_bootstrapper.py

and then

_select_pex_python_interpreter()

~/.pyenv/versions/3.7.5/bin/python3.7

doesn’t work, but absolute path works

The first is V1 code that is ignored by what you’re using. The second one is what is relevant. V2 code has a couple places where it influences how Pex chooses interpreters: 1) Where you discovered in hermetic_pex.py, which will influence what PEX sees on the PATH. In V2, subprocesses are almost entirely hermetic by default - they don’t leak things like your environment variables, unless we explicitly set that up. This is why something like

.python-version

will have no impact. 2) Interpreter constraints. See

pex.py

134-138 in Pants 3) Other options like

--python-setup-indexes

or

--not-zip-safe

. See

python_create_binary.py

L47-59 for some of these.

Following our discussion, I found when building the binary, the value set in

interpreter_constraints

is ignored, since it’s only used as an option to pex … (I deleted all versions in pyenv, and installed only 3.6.10, even I have constraints like

"CPython>=3.7",

, the binary could be built successfully) So, my current workable solution is having only 3.7.5 installed in pyenv, to make sure the PATH in pex subprocess only have that version presented. I don’t want to use

"<PEXRC>"

or

"<PATH>"

, which is a bit inconvenient … I’m wondering whether we could add a new

special_strings

in

expand_interpreter_search_paths

like

"<PYENV_LOCAL>"

, to let pants use the interpreter in

.python-version

of the

build_root

OR use

os.path.expandvars

before append the path into expanded ? https://github.com/pantsbuild/pants/blob/0d8fdf7c125c824828df3cbff7947af0193153c2/src/python/pants/python/python_setup.py#L244 I guess the second solution is more acceptable ? (and also very straightforward, but I might miss something)

Hi! Oh sorry, missed this when I woke up. Hm. Oh, wait, there might be a simpler solution. What happens if you set interpreter constraints to

CPython==3.7.5

? (Which, btw, you can use the simpler

==3.7.5

for the same thing) If that doesn’t work, the bigger issue is why interpreter constraints aren’t being used properly, rather than how to change the PATH to force it to use the one you want. -- Also, are you using

platforms

on this

python_binary

?

yes I’m using platform on this python binary, so interpreter constraint will not be passed right ?

Yes, that was the change from last week so that we stopped using the

current

platform when building, if you recall your question about why both macOS and linux wheels were being included in your binary. In https://github.com/pantsbuild/pex/issues/957, one of the authors of Pex, John, explains that uses

--interpreter-constraint

signals that you want to be able to locally build things, so it always uses

current

. Which

platform

are you using here? I don’t know Anaconda well, but I suspect

anaconda3-2019.10

works with the platform string you’re specifying, which surprises me.

I’m using

current

and

linux-x86_64-cp-37-cp37m

, the python interpreter is

3.7.5

(not anaconda, I don’t want

anaconda

to be used)

Hm, I think that would work but need to sanity check it’s safe. @witty-crayon-22786 do you see any issues with the above message to change how we interpret

--python-setup-interpreter-search-paths

? That does make it much more useful.

that sounds reasonable, yea.

Awesome. @bulky-evening-62934 would you be interested in putting up a pull request to change this? If so, see https://www.pantsbuild.org/howto_contribute.html for a guide on how to contribute PRs.

Sounds cool, I will do it

Wonderful. Let us know if you run into any issues getting this all set up. Thanks!

@hundreds-father-404 I sent a PR https://github.com/pantsbuild/pants/pull/9613