The pep hits all the bits I'm worried about / my Pex lockfile experiment implements. The only differences are bike sheds. I used json, they spec toml. I used a single file, they split per platform. If we do end up deciding to go with a Pex lockfile impl, I'll use that PEP format.

This dovetails well w/ realizing pip-compile has several deficiencies and that we may want Pex to generate lockfiles opened https://github.com/pantsbuild/pants/issues/12470 to centralize that discussion

I think mixing in PEP-665 in that ticket is a distraction though. The key thing is are we locking correctly. From there the locking data and format flow.

That's valid. This ticket was originally: pip-compile vs Pex, given the deficiencies we've found in pip-compile iiuc, the main deficiency I'm not including from the ticket is that pip-compile and requirements.txt style lockfile might not fully express if we're locking correctly? I had thought that is solved clunkily w/ adding environment markers, like we're doing manually with Black

To form a lock file that works for projects Pants allows you to express you need to support: 1. interpreter constraints 2. platforms 1 is interesting since its not discrete. Alot of discussion has been had about one, but I think folks are now convinced its a real problem. It requires the locker can at least post-process a resolve to figure out if the set of dists resolved work for all possible interpreters included by 1 and fail or warn if not. 2 is less interesting since its discrete, but it means we need to be able to resolve for a foreign platform and generate a lock.

(I'm not sure where you'd prefer this conversation be located)

It requires the locker can at least post-process a resolve to figure out if the set of dists resolved work for all possible interpreters included by 1 and fail or warn if not.

Well, or pessimistic generation, right? Yes, it has the crying wolf UX problem, but iiuc solves interpreter constraints in particular The thing we cannot solve is sys_platform env-markers, because we can't pessimistically generate on other platforms

2 is less interesting since its discrete, but it means we need to be able to resolve for a foreign platform and generate a lock.

yeah, pip-compile's support for that is broken atm

What is pessimistic generation?

Identify IC range is 3.6-3.9 so let's generate lockfile w/ 3.6, 3.7, 3.8, and 3.9. Then check for discrepancies and merge by adding env markers to requirements that are not universal (like the black lockfile is manually done)

What if you only have a 3.6 though?

Then you error/warn lockfile may not be valid. And the downside is we will cry wolf more times than we need to But from a correctness standpoint, I think it should handle ICs. It will not handle sys_platform

From a correctness standpoint you must error if you don't post-process. I had an example for Pants that proved that out. The resulting single-interpreter lockfile was missing two entries - so it wasn't actually a lock file.

Because we were not generating for multiple interpreters and then merging them. We only attempted one interpreter in the IC range Please take another look at https://github.com/pantsbuild/pants/pull/12371 and https://github.com/pantsbuild/pants/pull/12389 to better understand what I mean by pessimistic generation. It is all implemented expect for the merging piece

Ok. Gotcha.

Does pessimistic generation sound good from a correctness standpoint specifically for IC ranges? Again, it does have two known issues: 1) does not handle sys_platform env markers 2) bad UX because it cries wolf

-I think so. 1 though is a real issue and will lead to incorrect lock files in the wild when an org has more than one targeted sys_platform but only generates a lock file on one of them. That lock file can then be incomplete - so incorrect. We might decide to offer a solution involving the ability to create lock files on multiple machines and then merge them later into a complete, correct lock (in a file, in a dir (like the PEP, whatever mechanism).

Great, we are absolutely in agreement there. And fwict I think we will need to have that type of solution even with post-processing:

We might decide to offer a solution involving the ability to create lock files on multiple machines and then merge them later into a complete, correct lock

Because it's not possible to generate the lock on another platform. (This is different than how to handle the

--platform

pex/pip option) What post-processing will buy us is eagerly detecting when this is necessary, rather than the user being confused why the lockfile generated on my M1 doesn't work on your linux

Exactly. The post processing gives an unambiguous answer to "Is the lock complete?". We may need a merge feature to allow folks to deal with the answer "No.".

Great, absolutely agreed. And I will update the ticket when I get home about post processing with those suggestions you recommended to make it more precise So, as posited in my latest comment there, this is a priorities question imo: how much to value better UX over the engineering cost to implement. Where, better UX == eagerly warning when platforms are a problem, and not crying wolf when ICs are not a problem If we stick with pip-compile, the engineering cost is pretty high fwict. If pex generates lockfile, that cost is much lower Meaning, I think we can table this conversation until we decide soon on pip-compile vs pex for generation