https://pantsbuild.org/ logo
#general
Title
# general
r

rough-vase-83553

09/21/2022, 2:05 PM
I'm curious if there's a way to speed up lockfile generation or incrementally modify the lockfile, e.g. add a single new dependency or upgrade a single dependency version. It currently takes about 2 minutes just for my subproject to generate the lockfile, even when no changes have been made to any
python_requirements
. Setting up a virtual environment with our existing tooling takes no more than 10 seconds after dependencies have been downloaded.
e

enough-analyst-54434

09/21/2022, 2:48 PM
There is a way but we punted on it. The assumption was updating requirements is rare - surely O(days); so suffering a 2 minute resolve every small N days is not awesome but not horrible. Perhaps you change requirements more often? To flesh it out more - the fundamental tricky bit is Pants doesn't do mutation and fast incremental resolves using Pip require mutation. You have to build a venv with the current lock, run the lock resolve from within that venv, then extract the diff. All of this can be done, but its a decent chunk of work that has not been prioritized yet because of the assumption I led with above. Today, each lock operation is done from a fresh slate. Let me see if I can dig up an old issue for you to vote up / chime in on.
Ok, there is no ticket broken out. This ticket covered 2 cases - lock files and incremental resolves. It got closed after only implementing the former: https://github.com/pantsbuild/pants/issues/14127 As its thread concludes, a new issue should be opened when there is need to start work on the incremental part.
r

rough-vase-83553

09/21/2022, 3:34 PM
Gotcha. Yeah it's not too bad, although it's one clear area where Pants falls far behind our internal tooling. I think one main use case is adding a lock file rebuild to CI as a way to determine whether the lockfile should be rebuilt and committed to VCS. Although there may be a better way of doing this.
e

enough-analyst-54434

09/21/2022, 3:59 PM
There is, but Pants does not expose it. For example:
Copy code
jsirois@Gill-Windows:~ $ pex3 lock create pex==2.1.104 -olock.json --indent 2
jsirois@Gill-Windows:~ $ pex3 lock update --dry-run check lock.json 2>/dev/null
jsirois@Gill-Windows:~ $ echo $?
0
jsirois@Gill-Windows:~ $ cp lock.json lock.orig
jsirois@Gill-Windows:~ $ vi lock.json
jsirois@Gill-Windows:~ $ diff -u lock.orig lock.json
--- lock.orig   2022-09-21 08:58:39.646271351 -0700
+++ lock.json   2022-09-21 08:58:49.396271351 -0700
@@ -35,7 +35,7 @@
   "pip_version": "20.3.4-patched",
   "prefer_older_binary": false,
   "requirements": [
-    "pex==2.1.104"
+    "pex"
   ],
   "requires_python": [],
   "resolver_version": "pip-legacy-resolver",
jsirois@Gill-Windows:~ $ pex3 lock update --dry-run check lock.json 2>/dev/null
jsirois@Gill-Windows:~ $ echo $?
1
jsirois@Gill-Windows:~ $ pex3 lock update --dry-run display lock.json
Updates for lock generated by cp310-cp310-manylinux_2_35_x86_64
  Would update pex from 2.1.104 to 2.1.105
👀 1
8 Views