Background: We are stuck maintaining a legacy application that needs to run on a Linux x86 64 target. The python runtime we have to use is 3.8.16. The developers have a Mac based system to do builds. The question: Is there a way to generate a lockfile using a docker_environment or a way to pass in specific environment variables to the generate-lockfile goal? We are in a sticky situation where we have a library we are depending on that has a transitive dependency that only has a source distribution. The dependency hierarchy looks like this:

allennlp==0.9.0
-- thinc==7.1.0 (source only)
  -- numpy>1.20.0

The problem we are running into is that when we do a

generate-lockfile

the

thinc

library (which is a source distribution) builds itself using the

numpy 1.25.1

which is incompatible (by requiring a minimum python version of 3.9) with our targeted python runtime (3.8.16). Using a constraint only works for the top level dependencies being brought in. The moment a source distribution is brought in, a subprocess is started to to do the build which does not honor a constraint file from the parent process (top level). BUT Instead you have to declare a PIP_CONSTRAINT environment variable and this will get passed to the subprocess runtimes and use the constraints. Hence we would like to use a docker_environment to generate the lockfile. This would allow us to ensure that our dependencies only include libraries for the x86 64 Linux platform. A lot of these issues are specifically due to pip non allowing subprocess arguments, but by virtue of pants using pip here we are 😞

How do you configure your interpreter constraints in

pants.toml

(and elsewhere)? Theoretically, Pants/PEX should understand that, for instance, you're using 3.8 and choose appropriate dependencies based on that.

not a pex issue. It's when pex calls the pip install the subprocess fails. This is because the transitive dependency is scoped at >1.20.0 for numpy. They introduced a breaking change in 1.25.0 where it now requires python 3.9 to run. Additionally since we don't own the package with the broken constraint, we are stuck consuming it the way it is.

Yep, PEX (theoretically) understands the interpreter version and feeds that into the pip invocation to choose dependencies better.

yeah, but doing a setup.py (for these older projects) when resolving the dependencies all falls on pip to do it, not the pex builder.

Yeah, for sure... but PEX tries to guide pip to not choose a problematic dependency. Of course, I might be wrong here, we can work through this by confirming your interpreter constraints. What are they?

pants.toml:

[GLOBAL]
pants_version = "2.16.0"
pants_workdir = "%(buildroot)s/build/.pants.d"
pants_subprocessdir= "%(buildroot)s/build/.pids"
local_store_dir = "%(buildroot)s/build/lmdb_store"
named_caches_dir = "%(buildroot)s/build/named_caches"
backend_packages = [
    "pants.backend.python",
    "pants.backend.shell",
    "pants.backend.docker"
]

[source]
root_patterns = [
    "/support/*",
    "/try"
]

[python]
interpreter_constraints = [ "==3.8.*" ]
resolves = { try = "try/try.lock" }
resolves_to_constraints_file = { try = "try/constraints.txt" }
resolves_to_interpreter_constraints = { try = ["==3.8.*"] }
enable_resolves = true

[environments-preview.names]
python38_builder = "support/allen-srl-deps:python38_builder"
srl_generate_lockfile = "try:srl_generate_lockfile"

[anonymous-telemetry]
enabled = false

okay, yep, apparently the

python_version >= 3.9

constraint in Numpy 1.25 doesn't do what I expected, and/or the 2.16 version of Pants (and PEX) doesn't read it. Sorry. I reduced the example and reproduced the behaviour on my mac: https://gist.github.com/huonw/540f0a1acd9495737a186cf7c34e7376 I tried upgrading that example to pants 2.17 and

pants generate-lockfiles

seems to work (even without constraints files or the

numpy

python_requirement

, both which failed in 2.16). This suggests to me three options to investigate: 1. upgrade to 2.17 2. remain on 2.16, and see if upgrading PEX resolves it: (you can apply the

[pex-cli].version

and

known_version

settings from 2.17 to help https://www.pantsbuild.org/docs/reference-pex-cli#known_versions) 3. remain on 2.16/default PEX and see if using a newer pip_version helps: https://www.pantsbuild.org/v2.16/docs/reference-python#pip_version

I'll try the upgrade first as it seems like the way to go forward on this.

@broad-processor-92400 and @blue-city-97042 when locking, Pex does not necessarily run using an interpreter that satisfies the ICs. IOW, Python 2.7 can create a

3.11.*

lock just fine. So I wanted to clear up that apparent misunderstanding.

Now the outstanding question is whether one can do this in a docker_environment. I was under the impression that there may be some architecture specific optional dependencies which would not resolve on a Mac (when the build is intended for Linux).

Pex (and thus Pants) can cross-resolve. It can only not cross-build. So, if you have, say, Python 2.7 on an old x86_64 Mac you can resolve for Linux arm Python 3.11 as long as all transitive dependencies are either not platform specific, or, for those that are, are available as pre-built wheels for Linux arm CPython 3.11. You'd use complete_platforms on your pex_binary target to do this.

awesome info, thanks 🙂