https://pantsbuild.org/ logo
#general
Title
# general
g

gorgeous-winter-99296

03/10/2023, 8:18 AM
How do I specify the python version that Pex uses while building? I've switched our repo to minimum version 3.9 but some Pex targets still use 3.8 on CI.
Copy code
ProcessExecutionFailure: Process 'Building 9 requirements for ...foo.pex from the locks/pants.lock resolve: ...' failed with exit code 1.
[2023-03-09T21:49:32Z] stdout:
[2023-03-09T21:49:32Z]
[2023-03-09T21:49:32Z] stderr:
[2023-03-09T21:49:32Z] There was 1 error downloading required artifacts:
[2023-03-09T21:49:32Z] 1. emote-rl 23 from git+<https://github.com/EmbarkStudios/emote@ts/memory-logging-proxy-wrapper#egg=emote-rl>
[2023-03-09T21:49:32Z]     pid 2670012 -> /root/.cache/pants/named_caches/pex_root/venvs/eac612bf592de69ec2b624a7a30f7fec79e2c3a3/ddab8011daaee380698ac2fb9701af18c90c03f6/bin/python -sE /root/.cache/pants/named_caches/pex_root/venvs/eac612bf592de69ec2b624a7a30f7fec79e2c3a3/ddab8011daaee380698ac2fb9701af18c90c03f6/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --use-deprecated legacy-resolver --isolated -q --cache-dir /root/.cache/pants/named_caches/pex_root/pip_cache download --dest /root/.cache/pants/named_caches/pex_root/downloads/resolver_download.tbpzqg62/usr.bin.python3.8 --no-binary :all: --no-deps git+<https://github.com/EmbarkStudios/emote@ts/memory-logging-proxy-wrapper#egg=emote-rl> --index-url <https://pypi.org/simple/> --extra-index-url <https://download.pytorch.org/whl/cpu/> --retries 5 --timeout 15 exited with 1 and STDERR:
[2023-03-09T21:49:32Z] ERROR: Package 'emote-rl' requires a different Python: 3.8.5 not in '>=3.9'
Note that the resolver uses python3.8. I've added the following environment variables:
Copy code
PANTS_PYTHON_INTERPRETER_CONSTRAINTS="['==3.9.*']"
PY="python3.9"
PYTHON_BIN_NAME="python3.9"
and
Copy code
[python]
interpreter_constraints = ["==3.9.*"]
Not sure if this is a caching issue (due to changing version) or something I need to configure. This is on 2.15.0.
h

happy-kitchen-89482

03/10/2023, 3:01 PM
What do you have under
[python]
for your lockfile config, and did you regenerate your lockfile?
FWIW you shouldn't need any of those env vars, although it makes sense that you tried those!
g

gorgeous-winter-99296

03/10/2023, 3:09 PM
I have indeed regenerated my lockfile, tons of times! This is what I have right now:
h

happy-kitchen-89482

03/10/2023, 3:12 PM
Which python is pex itself running on? i.e., what is
/root/.cache/pants/named_caches/pex_root/venvs/eac612bf592de69ec2b624a7a30f7fec79e2c3a3/ddab8011daaee380698ac2fb9701af18c90c03f6/bin/python --version
?
Presumably 3.8 but let's double-check
Also, try using a more modern pip, to see if that helps:
Copy code
[python]
pip_version = "23.0.1"
(That "requires a different Python" error message is from pip.)
g

gorgeous-winter-99296

03/10/2023, 3:26 PM
Python 3.8.5, indeed. Will try pip version now!
That option doesn't exist in any version of Pants..?
e

enough-analyst-54434

03/10/2023, 5:05 PM
@gorgeous-winter-99296 it's in 2.16.0a0 but I don't think that will help. Is there any way to run CI with
-ldebug
so I can see the command line Pants is using to call Pex here?
g

gorgeous-winter-99296

03/10/2023, 5:07 PM
Sure thing; though I think the command line was above already? Or is that some other one?
h

happy-kitchen-89482

03/10/2023, 5:08 PM
Oh, sorry, for some reason I thought you mentioned being on 2.16
g

gorgeous-winter-99296

03/10/2023, 5:09 PM
Ah , I couldn't find it in the docs at all. 🙂 No; 2.15. I can try bumping once I've found the commandline @enough-analyst-54434 needs 🙂
e

enough-analyst-54434

03/10/2023, 5:10 PM
@gorgeous-winter-99296 that is not the command line Pants uses to call Pex, that is a Pex internal subprocess invocation later on down the line.
g

gorgeous-winter-99296

03/10/2023, 5:18 PM
Copy code
spawned local process as Some(584879) for Process { argv: ["/usr/bin/python3", "./pex", "--tmpdir", ".tmp", "--jobs", "1", "--python-path", "/usr/local/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "--output-file", "src.py.erupt.erupt.entry_points/server.pex", "--no-emit-warnings", "--requirements-pex", "local_dists.pex", "--interpreter-constraint", "CPython==3.9.*", "--entry-point", "erupt.entry_points.server", "--sources-directory=source_files", "atomicwrites==1.4.0", "bottle~=0.12.0", "coloredlogs~=15.0", "emote-rl[torch]@ git+<https://github.com/EmbarkStudios/emote@4c5b31753e7a497fa57ab59e13344468510c920c#egg=emote-rl>", "grpcio-health-checking>=1.40.0", "grpcio-reflection>=1.40.0", "grpcio>=1.40.0", "netifaces==0.11.0", "numpy~=1.20", "protobuf~=3.20.1", "ruamel.yaml~=0.16.0", "--lock", "locks/pants.lock", "--no-pypi", "--index=<https://pypi.org/simple/>", "--index=<https://download.pytorch.org/whl/cpu/>", "--manylinux", "manylinux2014", "--layout", "zipapp"], env: {"CPPFLAGS": "", "LANG": "en_US.UTF-8", "LC_ALL": "en_US.UTF-8", "LDFLAGS": "", "PATH": "/usr/local/bin:/usr/local/sbin:/usr/sbin:/usr/bin:/sbin:/bin", "PEX_IGNORE_RCFILES": "true", "PEX_PYTHON_PATH": "/usr/local/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "PEX_ROOT": ".cache/pex_root"}, working_directory: None, input_digests: InputDigests { complete: DirectoryDigest { digest: Digest { hash: Fingerprint<076529a5e38163646702e6698a95108391763f867f774443d7bcb46abb3c961e>, size_bytes: 417 }, tree: "Some(..)" }, nailgun: DirectoryDigest { digest: Digest { hash: Fingerprint<e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855>, size_bytes: 0 }, tree: "Some(..)" }, input_files: DirectoryDigest { digest: Digest { hash: Fingerprint<076529a5e38163646702e6698a95108391763f867f774443d7bcb46abb3c961e>, size_bytes: 417 }, tree: "Some(..)" }, immutable_inputs: {}, use_nailgun: {} }, output_files: {RelativePath("src.py.erupt.erupt.entry_points/server.pex")}, output_directories: {}, timeout: None, execution_slot_variable: None, concurrency_available: 11, description: "Building 11 requirements for src.py.erupt.erupt.entry_points/server.pex from the locks/pants.lock resolve: atomicwrites==1.4.0, bottle~=0.12.0, coloredlogs~=15.0, emote-rl[torch]@ git+<https://github.com/EmbarkStudios/emote@4c5b31753e7a497fa57ab59e13344468510c920c#egg=emote-rl>, grpcio-health-checking>=1.40.0, grpcio-reflection>=1.40.0, grpcio>=1.40.0, netifaces==0.11.0, numpy~=1.20, protobuf~=3.20.1, ruamel.yaml~=0.16.0", level: Info, append_only_caches: {CacheName("pex_root"): RelativePath(".cache/pex_root")}, jdk_home: None, platform: Linux_x86_64, cache_scope: Successful, execution_strategy: Local, remote_cache_speculation_delay: 0ns }
Copy code
ERROR: Package 'emote-rl' requires a different Python: 3.8.5 not in '>=3.9'
e

enough-analyst-54434

03/10/2023, 5:19 PM
Yeah, so the 1 VCS requirement. Ok, thanks.
And, last bit. Any chance I could get a copy of your lock file?
Ok, thank you. This is a Pex bug. I'll file shortly and probably have a fix out by morning California time.
g

gorgeous-winter-99296

03/10/2023, 5:57 PM
No rush, I won't look at this again until Monday morning CET ;-)
e

enough-analyst-54434

03/10/2023, 6:31 PM
Pretty embarrassing: https://github.com/pantsbuild/pex/issues/2092 This was just pure being dumb. Clearly you're the ~1st person using locks for VCS / local projects @gorgeous-winter-99296 and you're paying a price. Thanks for soldiering through being, what turns out, the tester for this functionality.
Alrighty, this should fix:
Copy code
[pex-cli]
version = "v2.1.129"
known_versions = [
  "v2.1.129|macos_arm64|717388fdf97eb6dad98fbe651debddfd05630aa6ce80557b8430efa9490fb7ec|4082068",
  "v2.1.129|macos_x86_64|717388fdf97eb6dad98fbe651debddfd05630aa6ce80557b8430efa9490fb7ec|4082068",
  "v2.1.129|linux_x86_64|717388fdf97eb6dad98fbe651debddfd05630aa6ce80557b8430efa9490fb7ec|4082068",
  "v2.1.129|linux_arm64|717388fdf97eb6dad98fbe651debddfd05630aa6ce80557b8430efa9490fb7ec|4082068"
]
g

gorgeous-winter-99296

03/13/2023, 10:37 AM
Aweome 🙇 Works perfectly!
e

enough-analyst-54434

03/23/2023, 4:40 PM
An interesting thing, the test added for this now fails on main and its due to a PDM release changing the hash of the wheel generated from the VCS requirement: https://github.com/EmbarkStudios/emote/blob/4c5b31753e7a497fa57ab59e13344468510c920c/pyproject.toml#L74-L75 I knew this was an issue already - locks lock dependencies but not build dependencies, but I'm now wondering if there is some sneaky way to solve this. IIUC there is no solution for this existing in the Python ecosystem.
The reason for the hash change is outside of PDM's control, its really a ~bad spec:
Copy code
$ unzip -qc emote_rl-23.0.0-py3-none-any.whl emote_rl-23.0.0.dist-info/WHEEL
Wheel-Version: 1.0
Generator: pdm-pep517 1.1.3
Root-Is-Purelib: True
Tag: py3-none-any
Generator is part of the spec IIRC.
Maybe I can just customize the hash for built wheels and filter out the Generator line.
g

gorgeous-winter-99296

03/23/2023, 4:48 PM
The version is optional per spec;
8.
Generator
is the name and optionally the version of the software that produced the archive.
e

enough-analyst-54434

03/23/2023, 4:49 PM
Hrm. Yeah, It does get in the way of reproducible builds.
If two different version of the build tool output the same bytes, save for that, it would be nice to hash the same.
But it would be better by far for the lock to contain meta-lock info for build tools.
g

gorgeous-winter-99296

03/23/2023, 4:55 PM
Would hashing just the files in the manifest be an option?
e

enough-analyst-54434

03/23/2023, 4:55 PM
METADATA is needed too. If Requires-* change, that should be reflected in the hash.
I'm not sure about others. It definitely gets finicky.
I think I'll just have to punt on this for now and update the expected test hash with a comment and wait for the next pdm release to break the test again. Maybe I come up with a solution between now and then.
g

gorgeous-winter-99296

03/23/2023, 5:20 PM
I had a peek at some backends (
hatchling
,
flit
,
build
,
pdm-backend
,
poetry-core
) and they all put the version in the wheel.
e

enough-analyst-54434

03/23/2023, 5:35 PM
Yeah. I think I need to think hard about how to lock PEP-518 build-requires in Pex locks.
5 Views