Will any of the future releases allow for utilizin...
# general
h
Will any of the future releases allow for utilizing the more incremental form of computing a lock file? I believe pex supports not recomputing from scratch each time.
b
Eventually. You wanna pick up the ticket? 😛
h
If it's an easy one, maybe I have some tinker time over vacation coming up
I'd hope it's something as simple as exposing an option in pants.toml and then triggering the right flags to lock generation
b
The update command requires additional args, so I think it'd be more along the lines of new CLI on the
lock
?
h
Ah right since you need to be able to specify packages you'd like to explicitly update (at least that's how pip-tools was)
b
Is there a ticket already written for this? If not, let's get one so that Nathanael and others have something to refer to.
b
e
There are two orthogonal things here. There is updating part of a lock which Pex supports via
pex3 lock update --lock my.lock -p project_with_security_fix -p project_with_security_bug<2.0.0+buggy_release
but Pants does not support. That leaves the lock untouched except for updating the listed
-p
projects and their transitive deps as needed if possible. This is generally useful to get bug fixes or security fixes as shown. Then there is updating a lock file, but faster than happens today. That also applies to the update form above which is also slow (under the covers it does a full re-lock pinning everything not in the
-p
set to what was in the lock already). That requires a new feature in Pex. I experimented a good while back and you get a good speedup by doing ~: 1. Create a venv from the full existing lock. 2. Run a full lock create (or update), but using a Pip that can see the venv with stuff in it. 3. Gather the delta. It's 2 that makes things faster for not exactly understood reasons. In short, Pip resolves faster that way / its algorithm shortcuts some stuff. That's the speed folks are used to with Pip since most people run Pip in a non-empty venv. Pex is currently hermetic and always runs Pip in an empty venv.
https://github.com/pantsbuild/pants/issues/15704 is not really this IIUC. Its more narrowly adding to a lock "Add ability to extend a PEX lockfile without modifying existing entries". That's not changing / updating versions, its more narrow.
@high-yak-85899 which of those 2 were you after? The speed incrementality or the update scope incrementality ... or both?
h
🔥 Speed 🔥
🚄 1
🐎 1
😂 1
🏎️ 1
Since we have a pretty big requirements file, adding a new requirement and triggering resolution of a bunch of packages takes more time than we'd like.
Happens infrequently, so not too big of an issue now.
e
Yeah, so the speed is the harder project.
It's #3 in that step list that's tricky.
Behind the scenes Pex uses
pip --log ...
to get a verbose log and scrapes that for resolve data. IIRC that log is different from what Pex currently expects in a normal full resolve; so there needs to be changes to log parsing / calculations performed using it + old lock data.
And, on the completeness front, IIRC there are tricky bits for local projects. People may or may not use local projects locking, but its there, and supporting speedy locks shouldn't break it.
I.E.: lock this source directory in its current state, not allowed to assume git, etc.