https://pantsbuild.org/ logo
Powered by
m

many-agent-62725

03/25/2021, 6:28 PM
Hello! I have a question regarding the lockfile (constraints.txt). We have the following configuration in the 
pants.toml
file:
Copy code
[python-setup]
interpreter_constraints = ["CPython>=3.8"]
requirement_constraints = "3rdparty/constraints.txt"
We also have the constraints file. In the CI pipeline, we are executing the following commands:
Copy code
./pants --version
./pants --changed-since=origin/master lint
./pants --changed-since=origin/master --changed-dependees=transitive test
However, when 
./pants --version
is executed, the lock file is not considered. Instead, the lock file is just used when 
./pants test
is being executed, according to the logs:
Copy code
./pants --changed-since=origin/master --changed-dependees=transitive test

[INFO] Starting: Resolving 3rdparty/constraints.txt
[INFO] Starting: Building pytest.pex with 3 requirements: pytest-cov>=2.10.1,<2.11, pytest>=6.0.1,<6.1, zipp==2.1.0
[INFO] Completed: Building pytest.pex with 3 requirements: pytest-cov>=2.10.1,<2.11, pytest>=6.0.1,<6.1, zipp==2.1.0
[INFO] Long running tasks:
 	Resolving 3rdparty/constraints.txt
[INFO] Long running tasks:
 	Resolving 3rdparty/constraints.txt
[INFO] Completed: Resolving 3rdparty/constraints.txt
This behaviour can be observed with the following dependency: 
constraints.txt
Copy code
cryptography==3.4.4
./pants --version
Copy code
Collecting cryptography>=1.3.4
    Downloading cryptography-3.4.7.tar.gz (546 kB)
Is this the expected behaviour? Because I was expecting the lock file to pin a specific version, so that we can have reproducible builds.
h

hundreds-father-404

03/25/2021, 6:31 PM
Hey @many-agent-62725, so those first few logs of
Collecting cryptography>=1.3.4
Downloading cryptography-3.4.7.tar.gz (546 kB) are what's used to bootstrap the virtualenv used to run Pants itself. That cryptography != the one used by your own code, only what is used to run Pants. This is all controlled by the 
./pants
bash script (which you can feel free to modify, including adding 
--constraints
to point to the line that says 
pip install
)
m

many-agent-62725

03/25/2021, 6:33 PM
image.png
Copy code
"${staging_dir}/install/bin/pip" install -U pip
This one here, right?
h

hundreds-father-404

03/25/2021, 6:34 PM
The following one with 
pants_requirement
m

many-agent-62725

03/25/2021, 6:35 PM
Oh, right, apologies 😅
--constraints 3rdparty/constraints.txt
I have another question as well: when executing 
pants package
(so as to build the AWS lambdas), will the constraints file be considered as well?
h

hundreds-father-404

03/25/2021, 6:37 PM
Something like that. I'm not certain if the relpath will work > when executing pants package (so as to build the AWS lambdas), will the constraints file be considered as well? Yep! When Pants is building for your own code, it will always use that constraints file. The sole time it's not being used is to bootstrap the Pants tool itself, which should be an implementation detail that doesn't impact your own code This is similar to how Pants requires Python 3.7 or 3.8 to run, but your own code can be whatever like Python 2.7
👍 1
m

many-agent-62725

03/25/2021, 6:47 PM
Copy code
--constraints 3rdparty/constraints.txt
Humm, it seems pip is complaining about the option
Copy code
"${staging_dir}/install/bin/pip" install --constraints 3rdparty/constraints.txt ${maybe_find_links} --progress-bar off "${pants_requirement}"
Should it be 
-c
?
Copy code
ERROR: Could not find a version that satisfies the requirement packaging==20.4 (from pantsbuild-pants)
ERROR: No matching distribution found for packaging==20.4
Now it’s a new one 😅
Interesting, the constraints.txt has this entry:
Copy code
packaging==20.9
Let’s see what happens if I change the version…
h

hundreds-father-404

03/25/2021, 6:53 PM
Yeah so taking a step back, I wouldn't recommend using your own code's constraints file for installing the Pants tool itself. There's generally not much reason to try to keep those two in sync, and it would be really frustrating for you to maintain. The deps Pants uses to run itself should have no bearing on your own code If you did want, you could have a dedicated constraints file for installing the Pants tool itself, but not sure you want to maintain that. It's probably something we should be maintaining as the project anyways.
Yeah so taking a step back
What's your objective with using the constraints file for Pants's own bootstrapping? It is a useful thing to ensure the resolve is always deterministic, but I'm trying to understand if there's additional motivation like avoiding a problematic dependency
m

many-agent-62725

03/25/2021, 6:55 PM
I don’t want to use a constraints file for when Pants is bootstrapping itself, however, it seems it’s including the dependencies we are using as well for some reason.
The issue we’re having is that the cryptography dependency is trying to compile itself with Rust compiler, but we don’t have permissions to use Rust (nor do we want to use it)
For this dependency in particular, it was updated 1 hour ago
which started causing the failure of the pipelines 😅
h

hundreds-father-404

03/25/2021, 6:58 PM
When is that Rust compilation happening? Is it when running 
./pants --version
the first time to bootstrap Pants itself?
m

many-agent-62725

03/25/2021, 6:58 PM
Yes
Another interesting thing I found is that: if I comment ./pants --version, the bdist version is used instead
Copy code
18:01:40  Collecting cryptography>=1.3.4
18:01:40    Downloading cryptography-3.4.7-cp36-abi3-manylinux2014_x86_64.whl (3.2 MB)
This was triggered by
Copy code
./pants --changed-since=origin/master lint
h

hundreds-father-404

03/25/2021, 7:01 PM
Okay, so then what you can do is a partial constraints file to pin only that dep. Have nothing more than 
cryptography==<version>
. Then wire it up to the Bash script To be clear, this indeed is an issue with the cryptography being used to install Pants itself. It is not related to the cryptography used by your own code
m

many-agent-62725

03/25/2021, 7:04 PM
OK, let me try that
OK, so it’s a dependency coincidence 😄
🙃 1
Copy code
"${staging_dir}/install/bin/pip" install -c 3rdparty/pants-constraints.txt ${maybe_find_links} --progress-bar off "${pants_requirement}"
🤞 1
Copy code
19:04:58  Collecting cryptography>=1.3.4
19:04:58    Downloading cryptography-3.4.4-cp36-abi3-manylinux2014_x86_64.whl (3.2 MB)
19:04:59  Collecting pyOpenSSL>=0.14
19:04:59    Downloading pyOpenSSL-20.0.1-py2.py3-none-any.whl (54 kB)
It seems it worked!
❤️ 1
So this means we need to take special care of this constraints file when upgrading 🤔
image.png
Is this file being used when bootstrapping Pants?
or the requirements.txt, I just wanted to know the requirements.txt you are using, so that I can upgrade it when I upgrade Pants
h

hundreds-father-404

03/25/2021, 7:12 PM
Oh that's an interesting idea, to somehow track that constraints file we maintain for the project. Like, if we could modify the 
./pants
script we distribute so that it pulls in that constraints file and uses it to have a more reproducible build when bootstrapping Pants But, no, you don't need to worry about staying in sync with 
pants/3rdparty/python/constraints.txt
- that is what we use internally when hacking on Pants, but it's not what the project requires. The project requires a subset of what's in 
requirements.txt
. The crytography requirement is a transitive dep of Pants, and note that it was floating of >=1.3.4
m

many-agent-62725

03/25/2021, 7:14 PM
Ohh, you are right, it’s a transitive one!
As for 
./pants
itself, it might be interesting to have a constraints file dedicated just for itself, otherwise, issues like this one might occur, where a transitive dependency is automatically updated 😅
In the meantime, I’ll keep an eye for this 
pants-constraints.txt
. And when upgrading, I might remove all constraints and add them one by one again
h

hundreds-father-404

03/25/2021, 7:18 PM
I agree, it would be good to have a constraints file for it. And my thinking is to track it against that one you discovered in pantsbuild/pants
Oh that's an interesting idea, to somehow track that constraints file we maintain for the project.
But the tricky part is versioning - the version of the constraints file you consume needs to be linked to your 
pants_version
. The more satisfying fix is to start releasing Pants as a Rust binary with all deps bundled in it: https://github.com/pantsbuild/pants/issues/7369
🤔 1
m

many-agent-62725

03/25/2021, 7:19 PM
OK, so have Rust execute Python (and not the other way around), is that so?
👍 1
I’m not familiar with Rust, but it’s a compiled language, correct? Which means you need to have multiple binaries for each architecture you want to support
h

hundreds-father-404

03/25/2021, 7:22 PM
Yeah, that is correct. Although we already must do that for pantsbuild.pants because it includes the Rust engine as a resource file, and that file only works with certain architectures. pantsbuild.pants is a "native" wheel, rather than a more conventional "pure Python" wheel
👍 1
m

many-agent-62725

03/25/2021, 7:23 PM
OK, I see. I just asked this because I don’t have full control over the pipeline, and binary dependencies are sometimes a headache to have them work 😅
But yeah, having Pants dependencies already bundled up avoids this kind of issues I’ve just had
➕ 1
Once again, thank you so much for your help 🙏
❤️ 1
e

enough-analyst-54434

03/25/2021, 7:27 PM
Is this just the issue that Pants itself uses a constraints "lockfile" but out setup-py packaging goal neglects to use this for install_requires?
h

hundreds-father-404

03/25/2021, 7:27 PM
Yes @enough-analyst-54434
e

enough-analyst-54434

03/25/2021, 7:27 PM
So that's a bug or missing feature to say please use it.
Then - I didn't follow all the constraints versioning stuff well - the versioning is simply the distribution itself - which is already versioned.
h

hundreds-father-404

03/25/2021, 7:28 PM
Yeah, good point. I'd classify as new feature - for a python_distribution, you may want to use your looser python_requirement_library targets. Depends if you're building a binary vs. library
e

enough-analyst-54434

03/25/2021, 7:29 PM
Exactly.
h

hundreds-father-404

03/25/2021, 7:29 PM
Then - I didn't follow all the constraints versioning stuff well - the versioning is simply the distribution itself - which is already versioned.
Yeah that was a very crude solution superseded by your much more elegant idea
e

enough-analyst-54434

03/25/2021, 7:29 PM
Elegant is a 4-letter word.
❓ 1
Powered by