https://pantsbuild.org/ logo
h

high-yak-85899

12/13/2022, 9:41 PM
Will any of the future releases allow for utilizing the more incremental form of computing a lock file? I believe pex supports not recomputing from scratch each time.
b

bitter-ability-32190

12/13/2022, 9:45 PM
Eventually. You wanna pick up the ticket? 😛
h

high-yak-85899

12/13/2022, 9:56 PM
If it's an easy one, maybe I have some tinker time over vacation coming up
I'd hope it's something as simple as exposing an option in pants.toml and then triggering the right flags to lock generation
b

bitter-ability-32190

12/13/2022, 9:59 PM
The update command requires additional args, so I think it'd be more along the lines of new CLI on the
lock
?
h

high-yak-85899

12/13/2022, 10:01 PM
Ah right since you need to be able to specify packages you'd like to explicitly update (at least that's how pip-tools was)
b

busy-vase-39202

12/13/2022, 10:03 PM
Is there a ticket already written for this? If not, let's get one so that Nathanael and others have something to refer to.
b

bitter-ability-32190

12/13/2022, 10:18 PM
e

enough-analyst-54434

12/13/2022, 10:18 PM
There are two orthogonal things here. There is updating part of a lock which Pex supports via
pex3 lock update --lock my.lock -p project_with_security_fix -p project_with_security_bug<2.0.0+buggy_release
but Pants does not support. That leaves the lock untouched except for updating the listed
-p
projects and their transitive deps as needed if possible. This is generally useful to get bug fixes or security fixes as shown. Then there is updating a lock file, but faster than happens today. That also applies to the update form above which is also slow (under the covers it does a full re-lock pinning everything not in the
-p
set to what was in the lock already). That requires a new feature in Pex. I experimented a good while back and you get a good speedup by doing ~: 1. Create a venv from the full existing lock. 2. Run a full lock create (or update), but using a Pip that can see the venv with stuff in it. 3. Gather the delta. It's 2 that makes things faster for not exactly understood reasons. In short, Pip resolves faster that way / its algorithm shortcuts some stuff. That's the speed folks are used to with Pip since most people run Pip in a non-empty venv. Pex is currently hermetic and always runs Pip in an empty venv.
https://github.com/pantsbuild/pants/issues/15704 is not really this IIUC. Its more narrowly adding to a lock "Add ability to extend a PEX lockfile without modifying existing entries". That's not changing / updating versions, its more narrow.
@high-yak-85899 which of those 2 were you after? The speed incrementality or the update scope incrementality ... or both?
h

high-yak-85899

12/13/2022, 10:38 PM
🔥 Speed 🔥
🚄 1
🐎 1
😂 1
🏎️ 1
Since we have a pretty big requirements file, adding a new requirement and triggering resolution of a bunch of packages takes more time than we'd like.
Happens infrequently, so not too big of an issue now.
e

enough-analyst-54434

12/13/2022, 10:42 PM
Yeah, so the speed is the harder project.
It's #3 in that step list that's tricky.
Behind the scenes Pex uses
pip --log ...
to get a verbose log and scrapes that for resolve data. IIRC that log is different from what Pex currently expects in a normal full resolve; so there needs to be changes to log parsing / calculations performed using it + old lock data.
And, on the completeness front, IIRC there are tricky bits for local projects. People may or may not use local projects locking, but its there, and supporting speedy locks shouldn't break it.
I.E.: lock this source directory in its current state, not allowed to assume git, etc.