👋 Hey all, I'm currently running into an issue regenerating subsystems pytest.lock where when I try to regenerate it with

./pants run build-support/bin/generate_builtin_lockfiles.py -- pytest

The pip_version changes to

20.3.4-patched

instead of changing to the expected

25.1.1

. This started happening midway during my development. I checked out the last commit I had where the lockfile had

"pip_version": "25.1.1"

, and tried regenerating the lockfile (without any code changes) - the

pip_version

changed to the patched version again, making me think something has changed within my dev environment. Any ideas what could be triggering this? generate_builtin_lockfiles.py sets --python-pip-version=latest, so surprised this isn't carrying over to the lockfile generation. cc @happy-kitchen-89482

Looking in

build-support/bin/generate_builtin_lockfiles.py

, it creates a temporary buildroot and runs from there (i.e., it does not use the main Pants

pants.toml

)

I'm having another engineer on my team run it in their env to see if they get the same output

Does hard-coding the pip version work? (versus the existing code setting it to

latest

)

Changing pip version doesn't work, and it looks like this code might be what is blocking it

Does turning on Pex debug logging reveal anything?

Yep, it makes sense that

20.3.4-patched

is the vendored version, but I'm not sure why two weeks ago I was seeing

25.1.1

as the pip version in the lockfile, and in prod the version is

24.2

so it seems like it's expected that the vendored version get's overriden in favor of latest version

I'm also curious what the exact Pex invocation is.

Let me add that, but I have been printing out what args get generated here and passed to pex3, here's what I'm getting:

['lock', 'create', '--tmpdir', '.tmp', '--no-emit-warnings', '--python-path', '/Users/ben.wilen/.pyenv/versions/2.7.18/bin:/Users/ben.wilen/.pyenv/versions/3.10.13/bin:/Users/ben.wilen/.pyenv/versions/3.11.9/bin:/Users/ben.wilen/.pyenv/versions/3.12.10/bin:/Users/ben.wilen/.pyenv/versions/3.8.18/bin:/Users/ben.wilen/.pyenv/versions/3.9.18/bin:/private/var/folders/t7/1d_xfv9s5nl33bs1sf_p5p1c0000gp/T/pants-sandbox-c2UKpM/.cache/pex_root/venvs/1/c103e30403fc818218c3301254360719661cbbea/5380d517d90ca2de95dd5b43313c6bedfe87c47f/bin:/Users/ben.wilen/.cache/pants/pants_dev_deps/fe2d7dc6846a5cbc52083d73263bec9cf29e69e3.venv/bin:/Users/ben.wilen/.cargo/bin:/Users/ben.wilen/.pyenv/shims:/opt/homebrew/opt/virtualenv/bin:/Users/ben.wilen/dd/data-eng-tools/bin:/Users/ben.wilen/go/bin:/Users/ben.wilen/dd/devtools/bin:/opt/homebrew/opt/coreutils/libexec/gnubin:/Users/ben.wilen/.rbenv/shims:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/usr/local/munki:/Users/ben.wilen/.volta/bin:/opt/homebrew/opt/scala@2.12//bin:/usr/local/src/spark-3.4.2-bin-hadoop3/bin', '--output=lock.json', '--style=universal', '--pip-version', 'latest', '--resolver-version', 'pip-2020-resolver', '--preserve-pip-download-log', 'pex-pip-download.log', '--target-system', 'linux', '--target-system', 'mac', '--indent=2', '--python-path=/Users/ben.wilen/.pyenv/versions/3.8.18/Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8', '--no-pypi', '--index=<https://pypi.org/simple/>', '--manylinux', 'manylinux2014', '--interpreter-constraint', 'CPython<4,>=3.8', 'pytest-cov!=2.12.1,<3.1,>=2.12', 'pytest-xdist<3,>=2.5', 'pytest==8.3.5']

What should the interpreter constraints be on the tool lockfiles?

Looks like everything is back to working! We could change interpreter constraints to 3.9+, but we'd also need to change that for pytest_extra_output_test.lock (and maybe any other downstream dependencies). Let me know what you think!

also, is there any negative effect from having the tool lockfiles generated with Pex's vendored pip?

Yeah, looks like that pip_version is incompatible with Python 3.12+

The Pip requested was pip@ git+<https://github.com/pex-tool/pip@6e5e5a40c9fc8622f5a3e4c9b173b018bf089cb9> but it does not work with the interpreter selected which is CPython 3.12.10 at /Users/ben.wilen/.pyenv/versions/3.12.10/bin/python3.12. Pip 20.3.4-patched requires Python <3.12.

Support for Python 3.8.x ended on 2024-10-07, so technically dropping support for it seems reasonable. (Although it should be a consensus decision of the maintainers.) In this case, the default tool lockfile could be 3.9 and any user with 3.8 would need to generate their own lock file.

Ok sounds good! Thanks for all your help debugging this