Hi pants team :wave: I'm in charge of python packa...
# welcome
m
Hi pants team šŸ‘‹ I'm in charge of python packaging at my workplace, and introduced pex to internally distribute a CLI application we developed. We're quite happy so far, but have run into this issue https://github.com/pantsbuild/pex/issues/539 which seems to have gone stale. It's blocking us, so I wanted to ask 1) if there is still progress being made here 2) if there is a workaround you could suggest and 3) if there is a way I could help with testing or developing
šŸ‘‹ 3
e
Arne, have you tried this with modern Pex? The Pex 2.x series vendors and uses Pip to do resolving; so proper abi3 handling and a host of other issues should have long since been fixed by that switch.
m
I wasn't aware of pex2, I'll check it out and report back
ok, I could isolate the issue a bit more. I was already using a recent enough version of pex. It looks like it's only an issue when using a local
--repo
and
--no-pypi
in conjunction with abi3 wheels
Copy code
# my env:
$ python --version
Python 3.9.7
$ pex --version
2.1.101
Copy code
# works
$ pex cryptography --platform linux_x86_64-cp-39-cp39 --output-file cryptography.pex
Copy code
# doesn't work
$ pip wheel cryptography -w wheelhouse
[...]
Saved ./wheelhouse/cryptography-37.0.4-cp36-abi3-manylinux_2_24_x86_64.whl
Saved ./wheelhouse/cffi-1.15.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Saved ./wheelhouse/pycparser-2.21-py2.py3-none-any.whl
$ poetry run pex cryptography --repo wheelhouse --no-pypi --platform linux_x86_64-cp-39-cp39 --output-file cryptography.pex
[...]
ERROR: Could not find a version that satisfies the requirement cryptography (from versions: none)
ERROR: No matching distribution found for cryptography
should I open an issue on gh?
e
Ah. I don't think so yet. What you need for situations like this is a
--complete-platform
, there is not enough information in
--platform
(an abbreviated platform) to fully characterize a foreign platform. See this documentation (or
pex --help
) to learn how to gather
--complete-platform
data: https://pex.readthedocs.io/en/v2.1.101/buildingpex.html#complete-platform
Another thing though, just to check, are you using
--platform
because, for example, you build the PEX on a Mac but want to run it on Linux? If not, you may not need to use
--platform
or
--complete-platform
at all.
m
yeah, we are building it on linux and want to support mac and more python versions than the one used to build the pex. I'll try to gather all complete-platform info we want to support.
Doesn't look too bad, but still a little debugging to do. I'm using the output from
pex3 interpreter inspect
which I passed a python3.8 and python3.9 interpreter as the
--complete-platform
argument (attached the file). I dropped the
--no-pypi
flag, since it seems to complicate things for little gain. The pex builds correctly, but: ā€¢ I needed to pre-build dependencies locally that only have source releases, since the
pex download
command that is run internally uses
--only-binary :all:
for some reason ā€¢ the interpreter I'm using to build the pex is 3.9, and it's not including binary wheels for 3.8:
Copy code
No interpreter compatible with the requested constraints was found:

  A distribution for cffi could not be resolved for /usr/local/bin/python3.8.
  Found 1 distribution for cffi that do not apply:
  1.) The wheel tags for cffi 1.15.1 are cp39-cp39-manylinux_2_17_x86_64, cp39-cp39-manylinux2014_x86_64 which do not match the supported tags of /usr/local/bin/python3.8:
  cp38-cp38-manylinux_2_31_x86_64
  ... 598 more ...
It's working if a 3.9 interpreter is available. I can't test on mac yet, but it would obviously not work yet because I don't know how to include it's info in the supported_platforms.json
e
since the
pex download
command that is run internally uses
--only-binary :all:
for some reason
That's because Pex has no magic to cross-build an sdist on Linux for Mac.
In order to use --platform or --complete-platform a pre-requisite is all required distributions must be available as pre-built wheels.
That complete platform file is very surprising. The tags indicate Python 3.9, the markers indicate Python 3.6 - that should not be possible. Can you provide the
pex3 interpreter inspect
command line(s) you use?
If you're targeting, say, 4 platforms, you should run the
pex3 interpreter inspect --python /this/python --markers --tags --indent 2 -o platform1.json
4 times, once on each platform / python you're targeting. Back in Pants you have 4 different
file(...)
targets.
m
whoops, I messed up in my initial call with the wrong python version. this is how I intended to run it:
pex3 interpreter inspect --markers --tags --python /home/arecknagel/pythons/pyenv/versions/3.8.12/bin/python --python /home/arecknagel/pythons/pyenv/versions/3.9.7/bin/python
, and then the resulting json is as follows, probably a more sane
e
More sane, but still you want just 1
--python
per file. So run that 2x, each with 1
--python
instead of 1x with 2
--python
like you did. That way you generate 2 different complete platform files.
Make sense?
šŸ‘ 1
Basically you replace each old abbreviated
--platform
with a
--complete-platform
file, one for one.
m
perfect, it works now! šŸŽ‰
e
Excellent.
m
thank you very much for walking me through this, I appreciate it.
e
You're welcome. Using --complete-platform is not simple to setup. It is easy to (re-)use after, but certainly not easy to get started with.