https://pantsbuild.org/ logo
Join Slack
Powered by
Another question regarding interpreter constraints...
# general
c
Another question regarding interpreter constraints (hazy area for me..)
Either I’m missing something, or there’s a missmatch between how the 
run
goal and the 
package
goal builds a pex file.
Copy code
$ ./pants run helloworld:
19:51:54.29 [INFO] Completed: Resolving constraints.txt
19:51:54.98 [INFO] Completed: Extracting 3 requirements to build pex_binary.pex from repository.pex: ansicolors==1.1.8, setuptools<57,>=56.2.0, types-setuptools<58,>=56.2.0
19:51:55.64 [INFO] Completed: Building local_dists.pex
Ahoj, pantsbuild!
vs.
Copy code
$ ./pants package helloworld:
19:52:10.24 [INFO] Wrote dist/helloworld/pex_binary.pex
$ ./dist/helloworld/pex_binary.pex 
Traceback (most recent call last):
  File "/x/.pex/unzipped_pexes/836fdc93bba6481e342cc3716bff839aab08f8dc/.bootstrap/pex/pex.py", line 475, in execute
    self.activate()
  File "/x/.pex/unzipped_pexes/836fdc93bba6481e342cc3716bff839aab08f8dc/.bootstrap/pex/pex.py", line 139, in activate
    self._activated_dists = self._activate()
  File "/root/.pex/unzipped_pexes/836fdc93bba6481e342cc3716bff839aab08f8dc/.bootstrap/pex/pex.py", line 126, in _activate
    activated_dists.extend(env.activate())
  File "/x/.pex/unzipped_pexes/836fdc93bba6481e342cc3716bff839aab08f8dc/.bootstrap/pex/environment.py", line 288, in activate
    self._activated_dists = self._activate()
  File "/x/.pex/unzipped_pexes/836fdc93bba6481e342cc3716bff839aab08f8dc/.bootstrap/pex/environment.py", line 632, in _activate
    resolved = self.resolve()
  File "/x/.pex/unzipped_pexes/836fdc93bba6481e342cc3716bff839aab08f8dc/.bootstrap/pex/environment.py", line 468, in resolve
    self._resolved_dists = self.resolve_dists(all_reqs)
  File "/x/.pex/unzipped_pexes/836fdc93bba6481e342cc3716bff839aab08f8dc/.bootstrap/pex/environment.py", line 489, in resolve_dists
    for qualified_req_or_not_found in self._root_requirements_iter(reqs):
  File "/x/.pex/unzipped_pexes/836fdc93bba6481e342cc3716bff839aab08f8dc/.bootstrap/pex/environment.py", line 433, in _root_requirements_iter
    raise ResolveError(message)
pex.environment.ResolveError: A distribution for setuptools could not be resolved in this environment.Found 1 distribution for setuptools that do not apply:
1.) The wheel tags for setuptools 56.2.0 are py3-none-any which do not match the supported tags of DistributionTarget(interpreter=PythonInterpreter('/usr/bin/python2.7', PythonIdentity('/usr/bin/python2.7', 'cp27', 'cp27mu', 'manylinux_2_31_x86_64', (2, 7, 18)))):
cp27-cp27mu-manylinux_2_31_x86_64
cp27-cp27mu-manylinux_2_30_x86_64
cp27-cp27mu-manylinux_2_29_x86_64
[...]
The 
pex_binary
interpreter_constraints is default
Copy code
interpreter_constraints
    type: Iterable[str] | None
    default: None
    The Python interpreters this code is compatible with.
and according to the help message, it will use that of 
[python].interpreter_constraints
which is:
Copy code
PANTS_PYTHON_INTERPRETER_CONSTRAINTS
  interpreter_constraints
      default: [
          "CPython>=3.6,<4"
      ]
      current value: [
          ">=3.7"
      ] (from pants.toml)
      The Python interpreters your codebase is compatible with.
So it is a mystery to me why package produced a pex that attempts to use the 2.7 version of python..
(This is all from the example-python repo)
There are these pythons on the system PATH
Copy code
$ python<TAB><TAB>
python              python2             python2.7           python3             python3.8           python3.8-config    python3-config      python3-futurize    python3-pasteurize  
$ python --version
Python 2.7.18
$ python3 --version
Python 3.8.10
w
hm… this might be a bug? if you crack open the PEX, does the PEX-INFO file show the constraints?
unzip -c $pexfile PEX-INFO | jq .
c
On my ubuntu it was 
unzip -p ..
😉 But no, the interpreter constraints field is
Copy code
"interpreter_constraints": [],
This could explain some of the confusion going around when trying to get pex’es to work with the 
experimental_shell_command
perhaps..
w
that’s very unexpected!
😮 1
c
Trying the same thing on my Mac, the ic is present in the `package`d pex as:
Copy code
"interpreter_constraints": [
    "CPython>=3.7"
  ],
However, executing it still fails:
Copy code
$ ./dist/helloworld/pex_binary.pex 
pyenv: python3.7: command not found

The `python3.7' command exists in these Python versions:
  3.7.12

Note: See 'pyenv help global' for tips on allowing both
      python2 and python3 to be found.
(using 
pyenv
) I tweaked the python search paths due to a bug (which I’m sure has been addressed, so not sure why it hit me now)
Copy code
$ PANTS_PYTHON_SETUP_INTERPRETER_SEARCH_PATHS='["<PYENV>"]' ./pants package helloworld:
To circumvent:
Copy code
ProcessExecutionFailure: Process 'Find interpreter for constraints: CPython>=3.7' failed with exit code 1.
...
  File "/Users/x/.cache/pants/named_caches/pex_root/installed_wheels/a62f822b62a14ee4072eb45dc8505f01ca805f7b/pex-2.1.51-py2.py3-none-any.whl/pex/interpreter.py", line 666, in create_interpreter
    raise cls.IdentificationError("Could not establish identity of {}.".format(binary))
pex.interpreter.IdentificationError: Could not establish identity of /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/pythonwrapper.app/Contents/MacOS/pythonwrapper.
here:
Copy code
$ python --version
Python 3.8.11
w
Copy code
However, executing it still fails:
in this case, the issue is probably the 
shebang
line (which can be overridden)… it’s odd that your pyenv doesn’t install a 
python3.7
alias… i thought that that was standard
the macOS vs Linux difference is very unexpected though
and yea, i don’t repro on macOS in a repo with global default interpreter constraints: they get passed through
c
OK, so the ubuntu and MacOS cases are most likely different issues. I’ll do some more digging, to see what comes up, will file issues on github for what I think will be addressable bugs.
❤️ 1
For the MacOS case, I don’t think I should need to override the shebang line here, as I ….. don’t have a py3.7 installed… eh ok, I think I see what could’ve tripped this up, heh 😛
Copy code
$ which python3.7
/Users/aadt/.pyenv/shims/python3.7
$ python3.7 --version
pyenv: python3.7: command not found

The `python3.7' command exists in these Python versions:
  3.7.12

Note: See 'pyenv help global' for tips on allowing both
      python2 and python3 to be found.
pyenv trying to be helpful, I think it should not unless invoked interactively…
w
since the shebang is just used to bootstrap (and pex will re-exec to a matching python if its doesn’t match), i’m not sure why we make it so specific.
i think maybe to avoid re-exec, which can take tens of milliseconds…? but
cc @enough-analyst-54434
c
heh 😛
e
Making it un-specific doesn't help. We have documented cases of no python (only python3) in the Debian family and vice versa. There are exactly 0 names for Python binaries you can rely on. This came up in another thread, but probably the only good answer is to force shebang to be specified. The user may be annoyed, but the user knows best.
c
I’m perfectly fine with it being specific, but it seems (pex? or pants) was fooled by the pyenv shim here, as I don’t have a py3.7 available.
e
That does not seem possible. At least on the Pex end the shebang is derived from a real Python interpreter being used in the building of the PEX.
c
OK, interesting. I’ll see if I can find proof, either way..
What’s also strange, is that 
./pants run
works, while 
./pants package
and then running the built pex file, does not work.
e
Ah, this is a bug in Pants 
<PYENV>
implementation I think. It exposes 
$(pyenv root)/versions/*/bin
but those are not on your 
PATH
for running the built PEX with, just the (potentially deactivated) shims.
👍 1
🎯 1
c
Yeah, I think you’re right… I do have a 3.7.12 version, but not on my PATH..
And for the 
run
goal, I get this:
Copy code
Completed: Find Python interpreter for constraints - Selected /Users/aadt/.pyenv/versions/3.7.12/bin/python3.7 to run PEXes with.
and well, the same for the 
package
goal,
so it seems the shebang is not specific enough for me, when it says:
Copy code
#!/usr/bin/env python3.7
as it doesn’t use the path where it found it…
e
If Pex used the full path then the PEX becomes unportable. So there are 18 evils to pick from.
c
Right.. very true. So it falls back to the pyenv usage again then. Two options that come to mind. A) Issue a warning if the selected python is not available on your PATH, or B) Only use.. oh wait, that's simply not using <PYENV> on you interpreter search path :p So a notice would be helpful, indicating that the selected version is not on the PATH.
e
Or we just fix how we implement<PYENV>.
👍 1
So, that warning will spam anyone building a PEX not for their machine. We could go into paragraph territory and note the PEX won't be directly runnable on this machine but it might on others... I continually dislike this sort of "helpful" advice that catches 80% of the cases or less or more, but others appear to like it. I guess it feels just like Stack Overflow education. It's generally partial and misleading and often damaging.
c
Mmm. I see your point. (and agree). And after sleeping another night with this added insight, the whole thing feels sort of dumb. I feel I could've realized much of the pyenv stuff sooner (and by myself). But I didn't, and I'm not sure of a proper approach to assist such cases in the future either. Fixing how PYENV is handled sounds promising (not entirely sure what that entails).
Oh, question: can I build a pex that's runnable with any python, say 3.6-3.10?
e
Yes: 1. Use 
--interpreter-constraint >=3.6,<3.11
2. Ensure you actually have 
python3.{6,7,8,9,10}
on your 
PATH
(or on 
--python-path
). 3. Pick a shebang you know works for the machines you want top run this on, say 
--python-shebang '/usr/bin/env python3'
👍 1
Step 2 is required if any of the requirements or their transitive dependencies are platform specific and not abi3.
You can always "manually" run the PEX after that if you want control over which of those Pythons is used; e.g.: 
python3.8 my_3.6-10.pex
will run the PEX with the 
python3.8
you chose since it matches interpreter constraints.
You can go further and make the PEX multi-platform with 
--platform
iff pre-built wheels are available for all foreign platforms you want to target.
c
Thanks John, will try this out on Monday <3
Open in Slack
PreviousNext
Powered by