https://pantsbuild.org/ logo
h

happy-kitchen-89482

07/30/2021, 1:44 AM
e

enough-analyst-54434

07/30/2021, 2:05 AM
The pep hits all the bits I'm worried about / my Pex lockfile experiment implements. The only differences are bike sheds. I used json, they spec toml. I used a single file, they split per platform. If we do end up deciding to go with a Pex lockfile impl, I'll use that PEP format.
h

hundreds-father-404

07/30/2021, 6:30 PM
This dovetails well w/ realizing pip-compile has several deficiencies and that we may want Pex to generate lockfiles opened https://github.com/pantsbuild/pants/issues/12470 to centralize that discussion
e

enough-analyst-54434

07/30/2021, 6:39 PM
I think mixing in PEP-665 in that ticket is a distraction though. The key thing is are we locking correctly. From there the locking data and format flow.
h

hundreds-father-404

07/30/2021, 6:43 PM
That's valid. This ticket was originally: pip-compile vs Pex, given the deficiencies we've found in pip-compile iiuc, the main deficiency I'm not including from the ticket is that pip-compile and requirements.txt style lockfile might not fully express if we're locking correctly? I had thought that is solved clunkily w/ adding environment markers, like we're doing manually with Black
sounds like it would help for me to purge PEP 665 from this ticket and decouple the lockfile format from this thread. I gtg to dentist, but will rewrite the ticket when I'm back
e

enough-analyst-54434

07/30/2021, 6:46 PM
To form a lock file that works for projects Pants allows you to express you need to support: 1. interpreter constraints 2. platforms 1 is interesting since its not discrete. Alot of discussion has been had about one, but I think folks are now convinced its a real problem. It requires the locker can at least post-process a resolve to figure out if the set of dists resolved work for all possible interpreters included by 1 and fail or warn if not. 2 is less interesting since its discrete, but it means we need to be able to resolve for a foreign platform and generate a lock.
h

hundreds-father-404

07/30/2021, 6:49 PM
(I'm not sure where you'd prefer this conversation be located)
It requires the locker can at least post-process a resolve to figure out if the set of dists resolved work for all possible interpreters included by 1 and fail or warn if not.
Well, or pessimistic generation, right? Yes, it has the crying wolf UX problem, but iiuc solves interpreter constraints in particular The thing we cannot solve is sys_platform env-markers, because we can't pessimistically generate on other platforms
2 is less interesting since its discrete, but it means we need to be able to resolve for a foreign platform and generate a lock.
yeah, pip-compile's support for that is broken atm
e

enough-analyst-54434

07/30/2021, 6:49 PM
What is pessimistic generation?
h

hundreds-father-404

07/30/2021, 6:52 PM
Identify IC range is 3.6-3.9 so let's generate lockfile w/ 3.6, 3.7, 3.8, and 3.9. Then check for discrepancies and merge by adding env markers to requirements that are not universal (like the black lockfile is manually done)
e

enough-analyst-54434

07/30/2021, 6:54 PM
What if you only have a 3.6 though?
That's the point of the post process, to prove you are capable of writing down a lockfile anyhow.
Along these lines, pip-compile just does this: https://github.com/jazzband/pip-tools/blob/b46a2e7370d832e00756448f952caa229a006611/piptools/repositories/base.py#L35-L41 I brought this up elsewhere, but it just grabs all hashes based on the 1 resolve using 1 inerpreter and whistles past the graveyard.
h

hundreds-father-404

07/30/2021, 6:57 PM
Then you error/warn lockfile may not be valid. And the downside is we will cry wolf more times than we need to But from a correctness standpoint, I think it should handle ICs. It will not handle sys_platform
e

enough-analyst-54434

07/30/2021, 6:58 PM
From a correctness standpoint you must error if you don't post-process. I had an example for Pants that proved that out. The resulting single-interpreter lockfile was missing two entries - so it wasn't actually a lock file.
(I'm not sure where you'd prefer this conversation be located)
You're probably right - this should move to the ticket.
h

hundreds-father-404

07/30/2021, 7:11 PM
Because we were not generating for multiple interpreters and then merging them. We only attempted one interpreter in the IC range Please take another look at https://github.com/pantsbuild/pants/pull/12371 and https://github.com/pantsbuild/pants/pull/12389 to better understand what I mean by pessimistic generation. It is all implemented expect for the merging piece
e

enough-analyst-54434

07/30/2021, 7:17 PM
Ok. Gotcha.
h

hundreds-father-404

07/30/2021, 8:00 PM
Does pessimistic generation sound good from a correctness standpoint specifically for IC ranges? Again, it does have two known issues: 1) does not handle sys_platform env markers 2) bad UX because it cries wolf
e

enough-analyst-54434

07/30/2021, 8:32 PM
-I think so. 1 though is a real issue and will lead to incorrect lock files in the wild when an org has more than one targeted sys_platform but only generates a lock file on one of them. That lock file can then be incomplete - so incorrect. We might decide to offer a solution involving the ability to create lock files on multiple machines and then merge them later into a complete, correct lock (in a file, in a dir (like the PEP, whatever mechanism).
1
h

hundreds-father-404

07/30/2021, 8:35 PM
Great, we are absolutely in agreement there. And fwict I think we will need to have that type of solution even with post-processing:
We might decide to offer a solution involving the ability to create lock files on multiple machines and then merge them later into a complete, correct lock
Because it's not possible to generate the lock on another platform. (This is different than how to handle the
--platform
pex/pip option) What post-processing will buy us is eagerly detecting when this is necessary, rather than the user being confused why the lockfile generated on my M1 doesn't work on your linux
e

enough-analyst-54434

07/30/2021, 8:44 PM
Exactly. The post processing gives an unambiguous answer to "Is the lock complete?". We may need a merge feature to allow folks to deal with the answer "No.".
1
h

hundreds-father-404

07/30/2021, 9:08 PM
Great, absolutely agreed. And I will update the ticket when I get home about post processing with those suggestions you recommended to make it more precise So, as posited in my latest comment there, this is a priorities question imo: how much to value better UX over the engineering cost to implement. Where, better UX == eagerly warning when platforms are a problem, and not crying wolf when ICs are not a problem If we stick with pip-compile, the engineering cost is pretty high fwict. If pex generates lockfile, that cost is much lower Meaning, I think we can table this conversation until we decide soon on pip-compile vs pex for generation