https://pantsbuild.org/ logo
h

hundreds-father-404

03/08/2022, 9:21 PM
Re 2.11 release timeline x Pex lockfiles:
Recap of the project status: ā€¢ 2.10 has a soft launch of multiple resolves. Feature gated because of issues w/ using Poetry for lockfile generation ā€¢ We just did 2.11.0.dev3, and should probably be doing rc0 this week. We stretched the release with 2.8 to dev5, and we reflected that it was too big of a release for our liking. ā€¢ John recently landed Pex support for consuming lockfiles, the main blocker to hooking up to Pants šŸ™Œ ā€¢ VCS support is WIP - it's a state-of-the-art feature and I imagine will take some time, and probably does not make sense to block rc0 on this.
Note that Pex not supporting VCS is the same as 2.10 not supporting it. That does not block changing from Poetry -> Pex. We still get several benefits by making the change in 2.11 So I'm wondering how does this sounds? ā€¢ Before rc0, change Pants to use Pex for lockfile generation & consumption. Possibly via deprecation - TBD ā€¢ The Python ecosystem is complex, I suspect dogfooding Pex lockfiles will reveal some edges not known about at https://github.com/pantsbuild/pex/projects/1. We can have a long rc period if necessary ā€¢ Plan to get VCS support in 2.12 or later
e

enough-analyst-54434

03/08/2022, 9:38 PM
I have no useful opinions about the labels (when branches are cut, etc). I do agree on qualitative timeline though: there will be Python ecosystem gotchas and this will take a while to suss out to something legitimately stable. So as long as we're not fooling people making them think ready by X.
šŸ‘ 1
Basically, locks have been a 1 man show so far and I'm a part-timer.
āž• 1
h

hundreds-father-404

03/08/2022, 9:45 PM
And I'm starting this afternoon on a time-sensitive Toolchain project that will take up most my time šŸ˜ John, I'm thinking to relieve pressure we can have three escape valves: 1. Poetry lockfile generation stays as an option. 2. An option I added last week
[python].resolve_generate_lockfiles
to allow you to manually manage your user lockfiles 3. We continue to not require using
[python].resolves
or
[tool].lockfile
. Not sure tho if we should deprecate
[python].requirement_constraints
Like 2.10 has a soft launch of "multiple resolves", 2.11 has a soft launch of Pex-generated lockfiles.
e

enough-analyst-54434

03/08/2022, 10:12 PM
I leave that all to you and others to decide. I refuse to feel pressure. Pex locks will be solid when they're solid and when I'm working I'm working on them hard.
šŸ‘ 1
Most of the pressure so far has been artificial fwict. We spoke a bit too soon.
h

hundreds-father-404

03/08/2022, 11:11 PM
https://github.com/pantsbuild/pants/issues/13964 summarizes some of this + has thoughts on how to wire up to Pants. fyi @bitter-ability-32190 - several of these tasks are highly parallelizable. tl;dr: ā€¢ add
[python].lockfile_generator
ā€¢ detect whether a lockfile is Pex JSON vs requirements.txt-style from Poetry ā€¢ strip JSON locks of Pants lockfile header (comments & JSON => boom) ā€¢ update
pex.py
to use
--lock
when appropriate Would you have bandwidth to take on any of those this week? I can try my best to fill in the rest so that we can get this into 2.11.0rc0
this week
@witty-crayon-22786 and I discussed there's also some work needed to stabilize
parametrize
, so rc0 this week is an aspiration and it's possible we slip to next week
b

bitter-ability-32190

03/08/2022, 11:27 PM
I can certainly pick something(s) up this week!
I'll read that issue tonight and check back in tomorrow
šŸ™Œ 1
ā¤ļø 2
h

hundreds-father-404

03/09/2022, 1:41 AM
I opened https://github.com/pantsbuild/pants/pull/14740 - this edit is so much smaller than I thought it might be šŸ™Œ @bitter-ability-32190 help appreciated if you have a spare moment to do any of the 3 TODOs in
pex.py
. They're kind of code-golfy ā›³ļø It would prob. be good to have a simple unit test for each. I can do the 2 TODOs in
lockfile.py
, along with anything you don't have time or interest in doing in
pex.py
šŸ™‚
e

enough-analyst-54434

03/09/2022, 2:00 AM
The 3rd TODO in
pex.py
is highly dubious. Pex absolutely make no guarantees about the format not changing and has a version field in there to allow for it. I'd address that todo last or else spend some time thinking about a sane way to not parse an undocumented format.
šŸ‘ 1
Also, couldn't the header just add a line about the lock type and then always strip the header? That would simplify I'd think. Any concerns about header stripping perf when the operation behind it is a resolve - these are slow no matter how fast, seems misguided.
h

hundreds-father-404

03/09/2022, 2:09 AM
The 3rd TODO in pex.py is highly dubious.
It was dubious to begin with...hm. The # of lines in a requirements.txt lockfile is not very helpful because
--hash
throws things off
Pex absolutely make no guarantees about the format not changing and has a version field in there to allow for it.
Good point. Hm.
couldn't the header just add a line about the lock type and then always strip the header
Yeah, but a key safety valve is
[python].invalid_lockfile_behavior
+
[python].resolves_generate_lockfiles
, which let you do manual lockfile generation. We cannot be certain a lockfile header is present in the lock. We probably want to drop that in Pants 2.12 or so, but until redesigned lockfiles are stable stable, this is a safety valve Maybe tho we can assume if you're using a manual lockfile, you're not using Pex format...that makes sense. This all is convoluted šŸ˜•
Any concerns about header stripping perf when the operation behind it is a resolve
Agreed
e

enough-analyst-54434

03/09/2022, 2:12 AM
Yeah, lots of complication from releasing a bit too early, but its a complicated project anyway.