public note-to-self: an important part of the spee...
# development
public note-to-self: an important part of the speedups that should be possible with PEX native lockfiles is skipping the building of the “repository pex”
, and directly consuming the lockfile to build each subset of the resolve. i haven’t followed up on this because there is a lot in flight, but doing that is likely something that would be worth getting into 2.11.x as a key benefit to the switch
👍 2
(as long as it isn’t risky? but i don’t see why it would be, relative to any other consumption of the lockfile)
I'm interested in taking a look at this. If you have any breadcrumbs, please leave them 🤓
@hundreds-father-404: i know you’re recovering (hope you feel better), but when you’re up to explaining this one that would be great
this would be great to implement. I have lots of doctors appointments / therapy still the next two weeks, so indeed your help @bitter-ability-32190 would be really helpful I think the main trick is
. That's where all the repository pex shenanigans happen.
is too general for that. My advice is to temporarily assume that all lockfiles are Pex format - how would we implement this? Now, still assume Pex format, but consider
or whatever it is. Finally, consider requirements.txt-style lockfiles which need to still do the repository Pex thing. It's gonna be pretty ugly code to figure this out, but I think worth it!
👍 1
Little PoC showing us sidestepping all non-veng pexs for deps
From cold cache, this takes
./pants lint ::
...and now we can incrementally lint a single file without paying the 10 minute penalty of a full repo PEX. This is glorious.
If @hundreds-father-404 and/or @witty-crayon-22786 take a quick look and confirm this the direction we want to head (stop using pex_path and put all deps hard/softlinked into final venv) I can round the PR out and make it nice, I'd very much like to have this in 2.11.x to roll out in an RC I can switch to in our repo, It'll be a huge win simple smile
WIP PR exists now, with metrics:
exciting! will definitely look this afternoon.
🙌 1
finally got around to it. thanks!
New pr being posted soon
awesome! thanks.