https://pantsbuild.org/ logo
#general
Title
# general
m

millions-kilobyte-51167

10/13/2021, 8:42 AM
Hi pants-people, I was invited into this slack due to https://github.com/pantsbuild/pex/issues/1477#issuecomment-941070905. After playing around a while with pex, I feel like I'm having some basic-understanding issues, since I'm running into issue after issue. My use case and assumptions: ā€¢ I want to build a pex file that contains a command line utility, and distribute it within my company for pipelines (debian) and humans (3+ linux archs and Mac) to use, targeting cpython 3.8, 3.9, and tentatively 3.10 ā€¢ I need fine-grained control over the dependencies, meaning that only 3rd party packages are used that were part of a successful test-suite exection (i.e., I run pex with
--no-pypi
and
--repo my_wheelhouse
) ā€¢ to populate
my_wheelhouse
, I build my application as a wheel, as well as all variants of my dependencies that I will need for the platforms I plan to support (plus
setuptools
, for some reason) ā€¢ since I am providing all the necessary dependencies, pex doesn't need to do run any build steps, so it doesn't matter which interpreter pex itself is installed with, or which ones it can see on the PATH Is everything ok so far? In particular the last point seems to be an issue, because my error messages change a lot depending on where pex was installed. I initially used
pipx
because it felt to me like a non-project bound build tool that I would want to install globally, and not into the virtualenv of my application. Similar to
poetry
, if that makes sense (by now, in my desperation, I have I think 4 installations of pex floating around on my system). I can try to build a reproducible example, but it would take me a while. I was hoping that someone can clear up for me how pex wants to be used instead šŸ˜ƒ
šŸ‘‹ 2
e

enough-analyst-54434

10/13/2021, 9:06 AM
Per the issue, what is the Pex command line you're using to create the PEX file with (key part being the list of
--platform X --platform Y ...
you use) vs the error you get when you try to run the PEX file on one of those platforms later?
How or where you install Pex the CLI tool should be irrelevant when passing
--platform
or
--python
or
--interpreter-constraint
. It only matters when you don't specify any of those.
m

millions-kilobyte-51167

10/13/2021, 9:24 AM
for now, my platform tags are just
--platform manylinux_2_27_x86_64-cp-39-cp39
and
--platform manylinux_2_27_x86_64-cp-38-cp38
, and I get errors during build. If I provide the dependencies as positional arguments (
pex my_app setuptools greenlet ...
) I get
Copy code
ERROR: Could not find a version that satisfies the requirement greenlet (from versions: none)
ERROR: No matching distribution found for greenlet
ERROR: Could not find a version that satisfies the requirement greenlet (from versions: none)
ERROR: No matching distribution found for greenlet
pid 22219 -> /home/me/.pex/venvs/4752d3a3799c096df952bdf2bf57e9caef61d7d8/4efcf8bbdd6ee48bfcee67ce2edeb12d1c553762/pex --disable-pip-version-check --no-python-version-warning --exists-action a --use-deprecated legacy-resolver --isolated -q --cache-dir /home/me/.pex --log /tmp/tmpeif168zc/pip.log download --dest /tmp/tmpi9aoh380/manylinux_2_27_x86_64-cp-39-cp39 --platform manylinux_2_27_x86_64 --implementation cp --python-version 39 --abi cp39 --only-binary :all: appdirs certifi charset-normalizer click dockerfile-parse gitdb gitpython greenlet idna pymysql python-gitlab requests-toolbelt requests semver six smmap sqlalchemy termcolor typing-extensions urllib3 setuptools my_application --no-index --find-links wheelhouse/ --retries 5 --timeout 15 exited with 1 and STDERR:
None
If I provide a requirements.txt with fixed versions, I get
Copy code
ERROR: Exception:
Traceback (most recent call last):
  File "/home/me/.pex/venvs/short/da72fb94/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 223, in _main
    status = self.run(options, args)
  File "/home/me/.pex/venvs/short/da72fb94/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 180, in wrapper
    return func(self, options, args)
  File "/home/me/.pex/venvs/short/da72fb94/lib/python3.8/site-packages/pip/_internal/commands/download.py", line 130, in run
    requirement_set = resolver.resolve(
  File "/home/me/.pex/venvs/short/da72fb94/lib/python3.8/site-packages/pip/_internal/resolution/legacy/resolver.py", line 170, in resolve
    requirement_set.add_requirement(req)
  File "/home/me/.pex/venvs/short/da72fb94/lib/python3.8/site-packages/pip/_internal/req/req_set.py", line 90, in add_requirement
    if not install_req.match_markers(extras_requested):
  File "/home/me/.pex/venvs/short/da72fb94/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 290, in match_markers
    return any(
  File "/home/me/.pex/venvs/short/da72fb94/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 291, in <genexpr>
    self.markers.evaluate({'extra': extra})
  File "/home/me/.pex/venvs/short/da72fb94/lib/python3.8/site-packages/pip/_vendor/packaging/markers.py", line 328, in evaluate
    return _evaluate_markers(self._markers, current_environment)
  File "/home/me/.pex/venvs/short/da72fb94/lib/python3.8/site-packages/pip/_vendor/packaging/markers.py", line 244, in _evaluate_markers
    lhs_value = _get_env(environment, lhs.value)
  File "/home/me/.pex/venvs/short/da72fb94/lib/python3.8/site-packages/pip/_vendor/packaging/markers.py", line 224, in _get_env
    raise UndefinedEnvironmentName(
pip._vendor.packaging.markers.UndefinedEnvironmentName: 'python_full_version' does not exist in evaluation environment.
ERROR: Exception:
Traceback (most recent call last):
  File "/home/me/.pex/venvs/short/da72fb94/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 223, in _main
    status = self.run(options, args)
  File "/home/me/.pex/venvs/short/da72fb94/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 180, in wrapper
    return func(self, options, args)
  File "/home/me/.pex/venvs/short/da72fb94/lib/python3.8/site-packages/pip/_internal/commands/download.py", line 130, in run
    requirement_set = resolver.resolve(
  File "/home/me/.pex/venvs/short/da72fb94/lib/python3.8/site-packages/pip/_internal/resolution/legacy/resolver.py", line 170, in resolve
    requirement_set.add_requirement(req)
  File "/home/me/.pex/venvs/short/da72fb94/lib/python3.8/site-packages/pip/_internal/req/req_set.py", line 90, in add_requirement
    if not install_req.match_markers(extras_requested):
  File "/home/me/.pex/venvs/short/da72fb94/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 290, in match_markers
    return any(
  File "/home/me/.pex/venvs/short/da72fb94/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 291, in <genexpr>
    self.markers.evaluate({'extra': extra})
  File "/home/me/.pex/venvs/short/da72fb94/lib/python3.8/site-packages/pip/_vendor/packaging/markers.py", line 328, in evaluate
    return _evaluate_markers(self._markers, current_environment)
  File "/home/me/.pex/venvs/short/da72fb94/lib/python3.8/site-packages/pip/_vendor/packaging/markers.py", line 244, in _evaluate_markers
    lhs_value = _get_env(environment, lhs.value)
  File "/home/me/.pex/venvs/short/da72fb94/lib/python3.8/site-packages/pip/_vendor/packaging/markers.py", line 224, in _get_env
    raise UndefinedEnvironmentName(
pip._vendor.packaging.markers.UndefinedEnvironmentName: 'python_full_version' does not exist in evaluation environment.
pid 10603 -> /home/me/.pex/venvs/4752d3a3799c096df952bdf2bf57e9caef61d7d8/4efcf8bbdd6ee48bfcee67ce2edeb12d1c553762/pex --disable-pip-version-check --no-python-version-warning --exists-action a --use-deprecated legacy-resolver --isolated -q --cache-dir /home/me/.pex --log /tmp/tmpv60fjclh/pip.log download --dest /tmp/tmpenn81s51/manylinux_2_27_x86_64-cp-39-cp39 --platform manylinux_2_27_x86_64 --implementation cp --python-version 39 --abi cp39 --only-binary :all: --requirement requirements.txt setuptools auxcli --no-index --find-links wheelhouse/ --retries 5 --timeout 15 exited with 2 and STDERR:
Failed to resolve for platform manylinux_2_27_x86_64-cp-39-cp39. Resolve requires evaluation of unknown environment marker: 'python_full_version' does not exist in evaluation environment.
e

enough-analyst-54434

10/13/2021, 9:34 AM
Concentrating on the 1st scenario, what are the wheel names you expect Pex to find to satisfy the greenlet dependency based on your
--index
and
--find-links
settings?
m

millions-kilobyte-51167

10/13/2021, 9:35 AM
greenlet-1.1.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
and
greenlet-1.1.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
I'm working on reproducing my problem in something I can upload to github, if the issue isn't trivial debugging like this is probably too much of a hassle
e

enough-analyst-54434

10/13/2021, 10:05 AM
I think you have enough info here actually. I've been silent but repro-ing.
Can you try using just 1 platform (for debugging less variables) and use
linux_x86_64-cp-39-cp39
?
For me, against pypi, that works until I run up against termcolor which only has an sdist there. You, though, presumably have a termcolor wheel in your wheelhouse.
m

millions-kilobyte-51167

10/13/2021, 10:12 AM
I'm getting
Failed to resolve for platform linux_x86_64-cp-39-cp39. Resolve requires evaluation of unknown environment marker: 'platform_machine' does not exist in evaluation environment.
for that, from
pip._vendor.packaging.markers.UndefinedEnvironmentName: 'platform_machine' does not exist in evaluation environment.
And yeah, I have a termcolor wheel in the local repo
e

enough-analyst-54434

10/13/2021, 10:12 AM
Thanks. Ok, let me dig into that a bit...
Ok, that's caused by SQLAlchemy requirements, namely:
Copy code
$ unzip -qc wheelhouse/SQLAlchemy-1.4.25-cp39-cp39-manylinux1_x86_64.manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_5_x86_64.whl SQLAlchemy-1.4.25.dist-info/METADATA | grep Requires-Dist | grep platform_machine
Requires-Dist: greenlet (!=0.4.17) ; python_version >= "3" and (platform_machine == "aarch64" or (platform_machine == "ppc64le" or (platform_machine == "x86_64" or (platform_machine == "amd64" or (platform_machine == "AMD64" or (platform_machine == "win32" or platform_machine == "WIN32"))))))
Fundamentally, Pex (and Pip) don't know the
platform_machine
for a foreign machine, and when you specify
--platform
you're effectively saying that platform is foreign. Double checking some code and the relevant PEP for the
platform_machine
env marker.
Ok, I think I can probably be filling in
platform_machine
for
--platform
resolves. That code is here: https://github.com/pantsbuild/pex/blob/c7e4921954418fdb018173b03a0bcb03b74ddc7b/pex/platforms.py#L264-L327 The one production use passes
default_unknown=False
very deliberately and that leads to the error you report: https://github.com/pantsbuild/pex/blob/da25bae605407b12461180fc8701cd0e78934b8e/pex/pip.py#L857-L861
šŸ‘€ 1
I wonder if that SQLAlchemy dep is right? For example, it will not use greenlet on mac silicon since that has a platform_machine of
arm64
.
m

millions-kilobyte-51167

10/13/2021, 11:09 AM
thanks a lot! I can help test the fix as soon as you've published the PR
e

enough-analyst-54434

10/13/2021, 11:24 AM
You're welcome. Should be up shortly...
šŸ™Œ 1
Ok @millions-kilobyte-51167 it's up here: https://github.com/pantsbuild/pex/pull/1489
Do you need instructions to get a Pex CLI running from that branch?
Just in case you can either run
python -mpex ...
from the root of the Pex repo or else
python -mpex . -cpex -opex
from the root of the Pex repo to create a Pex PEX that you can use as your Pex binary.
This has been odd hours for me in California and I have a big 9 hour drive coming up; so I won't be getting around to addressing further issues (if any) or doing a Pex 2.1.52 release with this fix until ~7PM Pacific time which will be around 14 hours from now.
m

millions-kilobyte-51167

10/13/2021, 12:01 PM
Thanks, I think I'll be able to manage from here on. Good luck on your trip!
e

enough-analyst-54434

10/14/2021, 2:40 AM
Alright @millions-kilobyte-51167 Pex 2.1.52 has the fix: https://pypi.org/project/pex/2.1.52/ Let me know if this gets you down the road to the finish line or you hit other issues.
m

millions-kilobyte-51167

10/14/2021, 10:16 AM
Alright, the build works now šŸŽ‰ When I run the resulting .pex with python3.9 on my path it works as expected, when I run it with 3.8 I get
Copy code
/usr/bin/env: 'python3.9': No such file or directory
The output of
pex-tools my_app.pex info
is
Copy code
{
  "bootstrap_hash": "3ec57293c54b7396d475ff4ae04db5ec823f6992",
  "build_properties": {
    "pex_version": "2.1.52"
  },
  "code_hash": "da39a3ee5e6b4b0d3255bfef95601890afd80709",
  "distributions": {
    "GitPython-3.1.24-py3-none-any.whl": "77355767101276ed6ceaf9ebf9606c995ce1672b",
    "PyMySQL-0.9.3-py2.py3-none-any.whl": "9cb76f040fbca52dafa4028b99b72f03189308d4",
    "SQLAlchemy-1.4.25-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "0772cc228abeac10caabc1aea12b3aa7eb917399",
    "SQLAlchemy-1.4.25-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "b560e2ff38b3d8ccf28e51db7ad83e61d428833c",
    "appdirs-1.4.4-py2.py3-none-any.whl": "b219560d4b28b017da90ce2c5484a2b6c62dfc52",
    "my_app-0.1.0-py3-none-any.whl": "6f0464af92d9ea860ecaf5c78b77106d8f18a629",
    "certifi-2021.10.8-py2.py3-none-any.whl": "ec6c8fc5615d73da818b9e3229f56b26c780d655",
    "charset_normalizer-2.0.6-py3-none-any.whl": "2d4f81e24c36da4736854f603fbc1f701e9dd144",
    "click-7.1.2-py2.py3-none-any.whl": "386b0f74344f046086c84bc4bc5c4f27f2f9a108",
    "dockerfile_parse-1.2.0-py2.py3-none-any.whl": "fcc88de95d27e7bf22517cdcff39dedaf39fed99",
    "gitdb-4.0.7-py3-none-any.whl": "cbb0a1a2c7f68bf77771d0c21546565b34ad655a",
    "greenlet-1.1.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "4493387f003343fbfedd8fd2042b7476868d2ebf",
    "greenlet-1.1.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "3973b70e466c646570645d00e65e28a8ec838eed",
    "idna-3.2-py3-none-any.whl": "f6f18646cd1fcc3b8d439bd113b7cc9ad4ebc3c0",
    "python_gitlab-2.10.1-py3-none-any.whl": "c8dfdc28961b02193158f9ba86c21cd7307100c8",
    "requests-2.26.0-py2.py3-none-any.whl": "fb70e1b8449a6408d983fb8c56f60f1e827890d8",
    "requests_toolbelt-0.9.1-py2.py3-none-any.whl": "1bbd7c07b41e671ad35e7784c56b5fa6ff9461f4",
    "semver-2.13.0-py2.py3-none-any.whl": "ccc5c528f6a241a427a8848b65617b6f4d652adc",
    "setuptools-58.2.0-py3-none-any.whl": "d025ab315362311e9bdb0d37e24d572e95f438e2",
    "six-1.16.0-py2.py3-none-any.whl": "035d7c208925c1832def39b592f3477ca36397bf",
    "smmap-4.0.0-py2.py3-none-any.whl": "3bc43d31c55385dd9f9f8ac4df4dc9575abb93f1",
    "termcolor-1.1.0-py3-none-any.whl": "ff40285b0a9a85789094b0aa04981f79386ef6f1",
    "typing_extensions-3.10.0.2-py3-none-any.whl": "e1248851ba7610196a1039644897ce611bd8536d",
    "urllib3-1.26.7-py2.py3-none-any.whl": "0468d103d09253d2e6af05983feaa1ab4d39394c"
  },
  "emit_warnings": true,
  "entry_point": "my_app.cli:main",
  "ignore_errors": false,
  "includes_tools": false,
  "inherit_path": "false",
  "interpreter_constraints": [],
  "pex_hash": "9f2024f13eb801f8fe1fda0b247ea9ed6dcc796c",
  "pex_path": null,
  "requirements": [
    "appdirs",
    "my_app",
    "certifi",
    "charset-normalizer",
    "click",
    "dockerfile-parse",
    "gitdb",
    "gitpython",
    "greenlet",
    "idna",
    "pymysql",
    "python-gitlab",
    "requests",
    "requests-toolbelt",
    "semver",
    "setuptools",
    "six",
    "smmap",
    "sqlalchemy",
    "termcolor",
    "typing-extensions",
    "urllib3"
  ],
  "strip_pex_env": true,
  "venv": false,
  "venv_bin_path": "false",
  "venv_copies": false,
  "pex_root": "/home/arecknagel/.pex"
}
e

enough-analyst-54434

10/14/2021, 10:32 AM
Yeah, Pex will not do magic with your shebang. You probably want to specify something like
--python-shebang '/usr/bin/env python'
or
--python-shebang /usr/bin/python3
and if either of those might pick a non-applicable python like python 3.7, you'll also need to specify something like
--interpreter-constraint '>=3.8,<3.10'
as well.
If it's not clear what's going on here with the shebang - a PEX is just a zip file with a shebang pre-pended; so the OS is launching the PEX using that shebang and Pex is not involved at that point. You have to craft a single shebang that works for all environments the PEX file will be deployed too.
m

millions-kilobyte-51167

10/14/2021, 10:41 AM
ok, I think that will work out just fine
only problem is how I can test both versions locally, but worst case I just copy them into a docker container with the python version I want to test against
e

enough-analyst-54434

10/14/2021, 10:45 AM
You can always run a PEX via
/usr/bin/python3.8 my.pex
and if the interpreter you picked is compatible with all
--interpreter-constraint
and
--python-path
(if any) you specified in the build, the PEX will run under that Python.
m

millions-kilobyte-51167

10/14/2021, 10:45 AM
ah, perfect.
e

enough-analyst-54434

10/14/2021, 10:47 AM
But you can't test macOS that way of course (in Linux CI for example). I curse macOS rules pertaining to vms.
m

millions-kilobyte-51167

10/14/2021, 10:49 AM
Not just you
Anyway, it looks like everything is working as expected right now. Thank you so much for the troubleshooting and the pex release, this will help me and my company a lot!
I appreciate your help to me and the community a lot, this has been a thoroughly enjoyable experience =)
3 Views