https://pantsbuild.org/ logo
Join Slack
Powered by
Is `PEX_PYTHON_PATH` solely meant to influence the...
# pex
h
Is 
PEX_PYTHON_PATH
solely meant to influence the interpreter used to run an interpreter, or also influence the interpreter used to build a PEX? Atm, it looks to only influence runtime
cc @chilly-magazine-21545 @happy-kitchen-89482
[python-setup].interpreter_search_paths
does not work if 
PEX_PYTHON_PATH
indeed should not impact build time. To influence the build time interpreter, you must set 
[pex].executable_search_paths
When 
internal_only=True
for a 
Pex
, we’re building a Pex for the sole purpose of choosing an interpreter based on constraints. Right now, it completely ignores 
[python-setup].interpreter_search_paths
and only uses the value from 
[pex].executable_search_paths
e
All pex env vars are only intended to influence runtime. The CLI has flags. The confusing bit is when the CLI itself is a PEX, which is the case for us, Pex PEX.
h
Oh, hi, I would love your guidance on this: https://pastebin.com/UseybiSR line 89 and 92 The user is trying to ensure Py37 is chosen from Pyenv. They set 
interpreter_search_paths
correctly to do this, which translates to 
PEX_PYTHON_PATH
. But Pex is still choosing macOS framework Py27 It appears 
PEX_PYTHON_PATH
is being ignored, and 
PATH
is being used in our rule 
find_interpreter() -> PythonExecutable
This makes sense to me. When building a PEX, the 
$PATH
is used for candidate interpreters, unless an RC file sets 
PEX_PYTHON_PATH
. I would think that the candidate interpreters need to be using the value of 
PEX_PYTHON_PATH
instead.
e
Copy code
spawned local process as 96891 for Process { argv: ["/usr/local/opt/pyenv/versions/3.7.7/bin/python", "./pex", "--interpreter-constraint", "CPython==2.7.*", "--interpreter-constraint", "CPython<3.9,>=3.4",
ICs say 2.7 is fine
h
Yes, but we instruct users that they should set 
[python-setup].interpreter_search_paths
to influence which interpreters can be used by the Python backend. That advice does not work here. The interpreter should both match the ICs, and be one of the allow-listed paths from 
interpreter_search_paths
User confirmed that changing 
[pex].executable_search_paths
fixed the issue: https://pantsbuild.slack.com/archives/C046T6T9U/p1602639174369900?thread_ts=1602173378.346900&amp;cid=C046T6T9U Meaning that changing the $PATH influences the interpreter selected, but changing $PEX_PYTHON_PATH has no impact for our 
find_interpreter
rule used by internal_only PEXes.
To avoid us having to make changes to Pex itself, Benjy’s idea was to leverage that PEX respects 
PEX_PYTHON_PATH
at buildtime when set in a rc file. We can have Pants create a temporary rc file whenever it executes a 
PexCliProcess
My original fix was to change Pex to respect 
PEX_PYTHON_PATH
at buildtime, given that it already respects it when used in rc files.
e
No pex env car should be respected at buildtime, buildtime should get a flag.
So, since PATH must be used to influence interpreter selection, we should be using interpreter_search_path to set PATH for the purposes of finding the interpreter to use to run internal only pexes.
That seems alot more direct than the rcfile thing.
h
Much simpler indeed, thanks for the recommendation. One thing I’m confused on, though: https://github.com/pantsbuild/pex/issues/1075 Why would the behavior matter if we’re running Pex as a Pex?
, since PATH must be used to influence interpreter selection, we should be using interpreter_search_path to set PATH for the purposes of finding the interpreter to use to run internal only pexes.
This will fix internal_only PEXes, but external PEXes like ones built for 
./pants package
will still not work properly. When creating the external Pexes, we need to set `$PATH == [pex].executable_search_paths`—and we can’t set `$PATH == [python-setup].interpreter_search_paths`—because creating the Pex may need binaries from the $PATH when building requirements. To robustly support both internal and external, I think we either need the RC file hack, or to change Pex’s behavior.
Open in Slack
PreviousNext
Powered by