https://pantsbuild.org/ logo
#pex
Title
# pex
h

hundreds-father-404

10/13/2020, 11:40 PM
Is
PEX_PYTHON_PATH
solely meant to influence the interpreter used to run an interpreter, or also influence the interpreter used to build a PEX? Atm, it looks to only influence runtime
cc @chilly-magazine-21545 @happy-kitchen-89482
[python-setup].interpreter_search_paths
does not work if
PEX_PYTHON_PATH
indeed should not impact build time. To influence the build time interpreter, you must set
[pex].executable_search_paths
When
internal_only=True
for a
Pex
, we’re building a Pex for the sole purpose of choosing an interpreter based on constraints. Right now, it completely ignores
[python-setup].interpreter_search_paths
and only uses the value from
[pex].executable_search_paths
e

enough-analyst-54434

10/14/2020, 1:21 AM
All pex env vars are only intended to influence runtime. The CLI has flags. The confusing bit is when the CLI itself is a PEX, which is the case for us, Pex PEX.
h

hundreds-father-404

10/14/2020, 1:24 AM
Oh, hi, I would love your guidance on this: https://pastebin.com/UseybiSR line 89 and 92 The user is trying to ensure Py37 is chosen from Pyenv. They set
interpreter_search_paths
correctly to do this, which translates to
PEX_PYTHON_PATH
. But Pex is still choosing macOS framework Py27 It appears
PEX_PYTHON_PATH
is being ignored, and
PATH
is being used in our rule
find_interpreter() -> PythonExecutable
This makes sense to me. When building a PEX, the
$PATH
is used for candidate interpreters, unless an RC file sets
PEX_PYTHON_PATH
. I would think that the candidate interpreters need to be using the value of
PEX_PYTHON_PATH
instead.
e

enough-analyst-54434

10/14/2020, 1:32 AM
Copy code
spawned local process as 96891 for Process { argv: ["/usr/local/opt/pyenv/versions/3.7.7/bin/python", "./pex", "--interpreter-constraint", "CPython==2.7.*", "--interpreter-constraint", "CPython<3.9,>=3.4",
ICs say 2.7 is fine
h

hundreds-father-404

10/14/2020, 1:33 AM
Yes, but we instruct users that they should set
[python-setup].interpreter_search_paths
to influence which interpreters can be used by the Python backend. That advice does not work here. The interpreter should both match the ICs, and be one of the allow-listed paths from
interpreter_search_paths
User confirmed that changing
[pex].executable_search_paths
fixed the issue: https://pantsbuild.slack.com/archives/C046T6T9U/p1602639174369900?thread_ts=1602173378.346900&amp;cid=C046T6T9U Meaning that changing the $PATH influences the interpreter selected, but changing $PEX_PYTHON_PATH has no impact for our
find_interpreter
rule used by internal_only PEXes.
To avoid us having to make changes to Pex itself, Benjy’s idea was to leverage that PEX respects
PEX_PYTHON_PATH
at buildtime when set in a rc file. We can have Pants create a temporary rc file whenever it executes a
PexCliProcess
My original fix was to change Pex to respect
PEX_PYTHON_PATH
at buildtime, given that it already respects it when used in rc files.
e

enough-analyst-54434

10/14/2020, 2:04 AM
No pex env car should be respected at buildtime, buildtime should get a flag.
So, since PATH must be used to influence interpreter selection, we should be using interpreter_search_path to set PATH for the purposes of finding the interpreter to use to run internal only pexes.
That seems alot more direct than the rcfile thing.
h

hundreds-father-404

10/14/2020, 2:09 AM
Much simpler indeed, thanks for the recommendation. One thing I’m confused on, though: https://github.com/pantsbuild/pex/issues/1075 Why would the behavior matter if we’re running Pex as a Pex?
, since PATH must be used to influence interpreter selection, we should be using interpreter_search_path to set PATH for the purposes of finding the interpreter to use to run internal only pexes.
This will fix internal_only PEXes, but external PEXes like ones built for
./pants package
will still not work properly. When creating the external Pexes, we need to set `$PATH == [pex].executable_search_paths`—and we can’t set `$PATH == [python-setup].interpreter_search_paths`—because creating the Pex may need binaries from the $PATH when building requirements. To robustly support both internal and external, I think we either need the RC file hack, or to change Pex’s behavior.