#21097 Allow way to pass build constraints to PEX Issue created by kkchau Is your feature request related to a problem? Please describe. Unpinned

build-system.requires

dependencies break over time. For example, we build a PEX using pants that installs

cnvkit

(https://github.com/etal/cnvkit), which depends on

pomegranate==0.14.9

(https://github.com/jmschrei/pomegranate/tree/v0.14.9), which in turn specifies

numpy

in its

build-system.requires

. `pomegranate`'s build process uses a deprecated (as of numpy 2.0.0, released a few days ago) API, which breaks when pants attempts to build it with the latest

numpy

release. Describe the solution you'd like We'd like a way to pass build constraints to the underlying PEX processes so that we can pin this build dependency (among others) from pants. Describe alternatives you've considered We've tried using the resolves_to_constraints_file directive to supply constraints to the PEX process, but that only pushes a

--constraints

flag (which in turn is pushed to the underlying

pip

processes), but this doesn't actually constraint build dependencies. As suggested in this thread, it seems like we need to specify the

PIP_CONSTRAINT

environment variable to actually constraint dependencies listed in

build-system.requires

since the constraints flag doesn't actually get passed to the build environment. Additional context Here's a minimal example:

# pants.toml
[GLOBAL]
pants_version = "2.21.0a0"
backend_packages = ["pants.backend.python"]

[pex-cli]
version = "v2.3.1"

[source]
root_patterns = ["/3rdparty"]

[python]
interpreter_constraints = ["==3.10.*"]
enable_resolves = true
resolves_generate_lockfiles = true
pip_version = "24.0"

[python.resolves]
cnvkit = "3rdparty/lockfiles/cnvkit.lockfile"

[python.resolves_to_interpreter_constraints]
cnvkit = ["==3.10.*"]

[python.resolves_to_constraints_file]
cnvkit = "3rdparty/constraints/cnvkit.constraints.txt"

# 3rdparty/cnvkit/BUILD.pants
python_requirement(
    name="lib",
    resolve="cnvkit",
    requirements=[
        "cnvkit@ git+<https://github.com/etal/cnvkit.git@bd533c33931a76b2373c4b053cdcb69c166b91b6>",
        "numpy==1.24",
        "pandas==1.5.1",
        "pomegranate==0.14.9",
    ],
)

pex_binary(
    dependencies=[":lib"],
    resolve="cnvkit",
)

# 3rdparty/constraints/cnvkit.constraints.txt
numpy<2.0.0
scipy<1.12.0

Output when running:

$ pants --keep-sandboxes=on_failure generate-lockfiles --resolve=cnvkit
09:09:19.11 [INFO] Preserving local process execution dir /private/var/folders/n_/c_8hhzf96f3d6h2nh529ybt40000gp/T/pants-sandbox-3MWj0u for Generate lockfile for cnvkit
09:09:19.11 [INFO] Completed: Generate lockfile for cnvkit
09:09:19.11 [ERROR] 1 Exception encountered:
Engine traceback:
  in `generate-lockfiles` goal

ProcessExecutionFailure: Process 'Generate lockfile for cnvkit' failed with exit code 1.
stdout:

stderr:
pid 89277 -> /Users/kevin.chau/.cache/pants/named_caches/pex_root/venvs/0deb89b23adbbad7a5b9aac06d2362acf7572cfa/0defb882924c4c7c56b6a0bfc3dce4b0af154aba/bin/python -sE /Users/kevin.chau/.cache/pants/named_caches/pex_root/venvs/0deb89b23adbbad7a5b9aac06d2362acf7572cfa/0defb882924c4c7c56b6a0bfc3dce4b0af154aba/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -q --cache-dir /Users/kevin.chau/.cache/pants/named_caches/pex_root/pip/24.0/pip_cache --log /private/var/folders/n_/c_8hhzf96f3d6h2nh529ybt40000gp/T/pants-sandbox-3MWj0u/.tmp/pex-pip-log.pbngi8de/pip.log download --dest /private/var/folders/n_/c_8hhzf96f3d6h2nh529ybt40000gp/T/pants-sandbox-3MWj0u/.tmp/tmphvs640fm/Users.kevin.chau..asdf.installs.python.3.10.14.bin.python3.10 --constraint 3rdparty/constraints/cnvkit.constraints.txt cnvkit@ git+<https://github.com/etal/cnvkit.git@bd533c33931a76b2373c4b053cdcb69c166b91b6> numpy==1.24 pandas==1.5.1 pomegranate==0.14.9 --index-url <https://pypi.org/simple/> --retries 5 --timeout 15 exited with 1 and STDERR:
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

# cd to sandbox and rerun command with increased verbosity for full output (add -vvvv in pants lock create command in __run.sh)
$ cd /private/var/folders/n_/c_8hhzf96f3d6h2nh529ybt40000gp/T/pants-sandbox-3MWj0u/
$ bash __run.sh
...
pid 97169 -> /private/var/folders/n_/c_8hhzf96f3d6h2nh529ybt40000gp/T/pants-sandbox-3MWj0u/.cache/pex_root/venvs/0deb89b23adbbad7a5b9aac06d2362acf7572cfa/0defb882924c4c7c56b6a0bfc3dce4b0af154aba/bin/python -sE /private/var/folders/n_/c_8hhzf96f3d6h2nh529ybt40000gp/T/pants-sandbox-3MW
j0u/.cache/pex_root/venvs/0deb89b23adbbad7a5b9aac06d2362acf7572cfa/0defb882924c4c7c56b6a0bfc3dce4b0af154aba/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -v --cache-dir /private/var/folders/n_/c_8hhzf96f3d6h2nh529ybt40000gp/T/pant
s-sandbox-3MWj0u/.cache/pex_root/pip/24.0/pip_cache --log /private/var/folders/n_/c_8hhzf96f3d6h2nh529ybt40000gp/T/pants-sandbox-3MWj0u/.tmp/pex-pip-log.ak2fam3j/pip.log download --dest /private/var/folders/n_/c_8hhzf96f3d6h2nh529ybt40000gp/T/pants-sandbox-3MWj0u/.tmp/tmpiug08bb0/Use
rs.kevin.chau..asdf.installs.python.3.10.14.bin.python3.10 --constraint 3rdparty/constraints/cnvkit.constraints.txt cnvkit@ git+<https://github.com/etal/cnvkit.git@bd533c33931a76b2373c4b053cdcb69c166b91b6> numpy==1.24 pandas==1.5.1 pomegranate==0.14.9 --index-url <https://pypi.org/simpl>
e/ --retries 5 --timeout 15 exited with 1 and STDERR:
...
  Collecting numpy
    Obtaining dependency information for numpy from <https://files.pythonhosted.org/packages/3b/61/e1e77694c4ed929c8edebde7d2ac30dbf3ed452c1988b633569d3d7ff271/numpy-2.0.0-cp310-cp310-macosx_14_0_arm64.whl.metadata>
    Using cached numpy-2.0.0-cp310-cp310-macosx_14_0_arm64.whl.metadata (60 kB)
    Link requires a different Python (3.10.14 not in: '>=3.7,<3.10'): https://files.pythonhosted.org/packages/99/f1/c00d6be56e1a718a3068079e3ec8ce044d7179345280f6a3f5066068af0d/scipy-1.6.2.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:>=3.7,<3.10)
    Link requires a different Python (3.10.14 not in: '>=3.7,<3.10'): https://files.pythonhosted.org/packages/fe/fd/8704c7b7b34cdac850485e638346025ca57c5a859934b9aa1be5399b33b7/scipy-1.6.3.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:>=3.7,<3.10)
    Link requires a different Python (3.10.14 not in: '>=3.7,<3.10'): https://files.pythonhosted.org/packages/bb/bb/944f559d554df6c9adf037aa9fc982a9706ee0e96c0d5beac701cb158900/scipy-1.7.0.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:>=3.7,<3.10)
    Link requires a different Python (3.10.14 not in: '>=3.7,<3.10'): https://files.pythonhosted.org/packages/47/33/a24aec22b7be7fdb10ec117a95e1e4099890d8bbc6646902f443fc7719d1/scipy-1.7.1.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:>=3.7,<3.10)
...
  Error compiling Cython file:
  ------------------------------------------------------------
  ...

  cpdef disable_gpu():
        global GPU
        GPU = False

  cdef ndarray_wrap_cpointer(void* data, numpy.npy_intp n):
      ^
  ------------------------------------------------------------

^ fails due to numpy 2.0.0 even though we specify constraints. Adding

PIP_CONSTRAINT

as

env

to

__run.sh

doesn't work either. What does work is rerunning the failed … pantsbuild/pants