many-agent-62725
03/25/2021, 6:28 PMpants.toml
file:
[python-setup]
interpreter_constraints = ["CPython>=3.8"]
requirement_constraints = "3rdparty/constraints.txt"
We also have the constraints file.
In the CI pipeline, we are executing the following commands:
./pants --version
./pants --changed-since=origin/master lint
./pants --changed-since=origin/master --changed-dependees=transitive test
However, when ./pants --version
is executed, the lock file is not considered. Instead, the lock file is just used when ./pants test
is being executed, according to the logs:
./pants --changed-since=origin/master --changed-dependees=transitive test
[INFO] Starting: Resolving 3rdparty/constraints.txt
[INFO] Starting: Building pytest.pex with 3 requirements: pytest-cov>=2.10.1,<2.11, pytest>=6.0.1,<6.1, zipp==2.1.0
[INFO] Completed: Building pytest.pex with 3 requirements: pytest-cov>=2.10.1,<2.11, pytest>=6.0.1,<6.1, zipp==2.1.0
[INFO] Long running tasks:
Resolving 3rdparty/constraints.txt
[INFO] Long running tasks:
Resolving 3rdparty/constraints.txt
[INFO] Completed: Resolving 3rdparty/constraints.txt
This behaviour can be observed with the following dependency:
constraints.txt
cryptography==3.4.4
./pants --version
Collecting cryptography>=1.3.4
Downloading cryptography-3.4.7.tar.gz (546 kB)
Is this the expected behaviour? Because I was expecting the lock file to pin a specific version, so that we can have reproducible builds.hundreds-father-404
03/25/2021, 6:31 PMCollecting cryptography>=1.3.4Downloading cryptography-3.4.7.tar.gz (546 kB) are what's used to bootstrap the virtualenv used to run Pants itself. That cryptography != the one used by your own code, only what is used to run Pants. This is all controlled by the
./pants
bash script (which you can feel free to modify, including adding --constraints
to point to the line that says pip install
)many-agent-62725
03/25/2021, 6:33 PM"${staging_dir}/install/bin/pip" install -U pip
hundreds-father-404
03/25/2021, 6:34 PMpants_requirement
many-agent-62725
03/25/2021, 6:35 PM--constraints 3rdparty/constraints.txt
pants package
(so as to build the AWS lambdas), will the constraints file be considered as well?hundreds-father-404
03/25/2021, 6:37 PMmany-agent-62725
03/25/2021, 6:47 PM--constraints 3rdparty/constraints.txt
"${staging_dir}/install/bin/pip" install --constraints 3rdparty/constraints.txt ${maybe_find_links} --progress-bar off "${pants_requirement}"
-c
?ERROR: Could not find a version that satisfies the requirement packaging==20.4 (from pantsbuild-pants)
ERROR: No matching distribution found for packaging==20.4
packaging==20.9
hundreds-father-404
03/25/2021, 6:53 PMYeah so taking a step backWhat's your objective with using the constraints file for Pants's own bootstrapping? It is a useful thing to ensure the resolve is always deterministic, but I'm trying to understand if there's additional motivation like avoiding a problematic dependency
many-agent-62725
03/25/2021, 6:55 PMhundreds-father-404
03/25/2021, 6:58 PM./pants --version
the first time to bootstrap Pants itself?many-agent-62725
03/25/2021, 6:58 PM18:01:40 Collecting cryptography>=1.3.4
18:01:40 Downloading cryptography-3.4.7-cp36-abi3-manylinux2014_x86_64.whl (3.2 MB)
./pants --changed-since=origin/master lint
hundreds-father-404
03/25/2021, 7:01 PMcryptography==<version>
. Then wire it up to the Bash script
To be clear, this indeed is an issue with the cryptography being used to install Pants itself. It is not related to the cryptography used by your own codemany-agent-62725
03/25/2021, 7:04 PM"${staging_dir}/install/bin/pip" install -c 3rdparty/pants-constraints.txt ${maybe_find_links} --progress-bar off "${pants_requirement}"
19:04:58 Collecting cryptography>=1.3.4
19:04:58 Downloading cryptography-3.4.4-cp36-abi3-manylinux2014_x86_64.whl (3.2 MB)
19:04:59 Collecting pyOpenSSL>=0.14
19:04:59 Downloading pyOpenSSL-20.0.1-py2.py3-none-any.whl (54 kB)
hundreds-father-404
03/25/2021, 7:12 PM./pants
script we distribute so that it pulls in that constraints file and uses it to have a more reproducible build when bootstrapping Pants
But, no, you don't need to worry about staying in sync with pants/3rdparty/python/constraints.txt
- that is what we use internally when hacking on Pants, but it's not what the project requires. The project requires a subset of what's in requirements.txt
. The crytography requirement is a transitive dep of Pants, and note that it was floating of >=1.3.4many-agent-62725
03/25/2021, 7:14 PM./pants
itself, it might be interesting to have a constraints file dedicated just for itself, otherwise, issues like this one might occur, where a transitive dependency is automatically updated 😅pants-constraints.txt
. And when upgrading, I might remove all constraints and add them one by one againhundreds-father-404
03/25/2021, 7:18 PMOh that's an interesting idea, to somehow track that constraints file we maintain for the project.But the tricky part is versioning - the version of the constraints file you consume needs to be linked to your
pants_version
.
The more satisfying fix is to start releasing Pants as a Rust binary with all deps bundled in it: https://github.com/pantsbuild/pants/issues/7369many-agent-62725
03/25/2021, 7:19 PMhundreds-father-404
03/25/2021, 7:22 PMmany-agent-62725
03/25/2021, 7:23 PMenough-analyst-54434
03/25/2021, 7:27 PMhundreds-father-404
03/25/2021, 7:27 PMenough-analyst-54434
03/25/2021, 7:27 PMhundreds-father-404
03/25/2021, 7:28 PMenough-analyst-54434
03/25/2021, 7:29 PMhundreds-father-404
03/25/2021, 7:29 PMThen - I didn't follow all the constraints versioning stuff well - the versioning is simply the distribution itself - which is already versioned.Yeah that was a very crude solution superseded by your much more elegant idea
enough-analyst-54434
03/25/2021, 7:29 PM