https://pantsbuild.org/ logo
Join Slack
Powered by
I’m interested in trying to tackle <https://github...
# development
b
I’m interested in trying to tackle https://github.com/pantsbuild/pants/issues/15704 over the holidays. I know incremental lockfile updates have been a long-desired feature in Pants, so I want to take a crack at making it happen. Is there anything I should know before trying this / any known dead ends?
c
The part I was stuck on was the groundwork to have "sub commands" so that we could have, ex: • 
pants lock generate
# everything type of lockfile supports this • 
pants lock sync
# specific to pex lockfiles, be a thin wrapper and avoid the 200% problem • 
pants lock some-other-pex-verb
https://github.com/pantsbuild/pants/issues/15568 https://github.com/pantsbuild/pants/issues/13694 I think maybe @happy-kitchen-89482 mentioned with the new option subsystem that was more viable now?
FWIW Here were some internal notes I had drawn up ### M0: Confirm sub cmds/goals viable(2-3 day-points) Right now all pants commands are in the form “pants goal –flags”. Can we do “pants goal sub-thingy –flags”? Is multiple top level goals an okay fallback? Why: Unlikely for us to be happy or accepted if you need a bazillion nested context flags to do anything. Risk: Possibly a deep rooted assumption. Worklog: ### M1: write out proposed pex→pants command mapping (~1 day-points) Unless I messed up pretty badly on review Pex should be able to do everything we need. Pex has lock {create,update,sync} with many sub-options. What’s the right minimal useful mapping to Pants? (hint: thin as possible) Why: Measure twice, cut once. Worklog: ### M2: generate-lockfile → lock-generate (~1 day-point) Assuming success above, make room for new commands. Worklog: ### M3: update (within existing range) (2-3 day-points) Subject to M1, update within range is likely something Worklog: ### M3: pex sync (2-4 day-points) Subject to M1, part of the answer is probably going to take the form of “use the pex command that looks at reqs.txt and makes the minimal changes”. Worklog:
b
Ok thanks, those are pretty useful notes. I was actually targeting a smaller changeset - rather than trying to mess with the subcommand system (a good rearchitecture IMO), I was just looking to add a new 
update-lockfiles
goal similar to 
generate-lockfiles
that could analogously do incremental updates for a single resolve.
c
oh hmm, oh that would be a reasonable way to unblock on that part. I like that So, something like 
experimental-sync-lockfiles
which would map to 
pex3 lock sync
bikeshed: maybe have 
pex
in those names? (I forget if 
sync
handles both the "update from the existing inputs" and "update within the existing range" use cases) I recommend not exposing the pex "sharp knife" variant
👍 1
h
Keep in mind that 
pex3 lock sync
is not faster than regenerating a full lockfile, it still does a full resolve, but applies constraints from the previous lockfile so that the actual updates are conservative. See https://github.com/pex-tool/pex/discussions/2565 for John patiently explaining this to me. So while I think this is an important change and would be happy to help guide you through it, keep in mind that it won’t improve lock performance.
And re subcommands, one way I have been experimenting with those in the new python backend experiments has been to use mutually-exclusive BoolOptions on the 
lock
goal, So you have 
lock --generate
, 
lock --sync
etc
b
Thanks for the link and totally understood on performance of 
pex3 lock sync
. The rough outline of the change I had in mind would actually address a different use case, and hopefully one where lock performance would be improved: updating or deleting dependencies from a lockfile. My idea here was as follows: 1. Create a new goal called 
update-lockfile-dependencies
and allow specifying a resolve. 2. For this resolve, compute the set of input package requirements based on generated and explicit 
python_requirements
targets, and diff that against the lockfile metadata to find any changes. a. Not sure how feasible this is. b. Meta-goal here is to overcome the constraint outlined in https://github.com/pantsbuild/pants/pull/20364/files#r1451604634: Pants combines Python dependencies from a variety of sources and it might not be easy to update the source file that Pants depends on for a particular requirement. 3. Then, use the replace/delete functionality introduced in PEX 2.1.160 to delete and then replace the appropriate dependencies from the lockfile.
Although thanks to this, I realize my idea won’t work:
Every lock subcommand (create, update, sync) currently performs a complete fresh lock from scratch. The only difference is that update and sync apply constraints based on a prior lock. The 
pex3 lock {update,sync}
commands get you incremental lock updates. They have 0 to do with speed of locking.
13 Views
Open in Slack
PreviousNext
Powered by