https://pantsbuild.org/ logo
Join Slack
Powered by
In other deficiencies for pip-compile, I can't get...
# development
h
In other deficiencies for pip-compile, I can't get it to generate for a foreign platform. This does not error 
./pip_compile.pex --pip-args '--platform=fake'
, suggesting the arg gets ignored. (But weirdly, 
--pip-args '--not-an-opt'
does fail)
I'm trying to still reason about how important this is.. The problem is that sometimes resolves are different on macOS vs Linux. A way to generate a lockfile for both platforms from a single machine is something like 
--platform
Otherwise, I think you'd need to generate the lockfile on both a macOS and a Linux machine, and (manually?) merge the two lockfiles by using environment markers to handle discrepencies, a la https://github.com/pantsbuild/pants/pull/12362
A way to generate a lockfile for both platforms from a single machine is something like --platform
But then this fails if you don't already have bdist_wheels for every requirement on the foreign platform 🤔
w
yes, but that’s the status quo with 
pex_binary
you must have wheels for the other platform.
and that doesn’t need to change here.
h
only if you opted into 
platforms
I'm talking about, say, generating Black lockfile. No 
platforms
are hardcoded. To be sure the lockfile is truly compatible w/ both OSes, we need to either pessimistacally generate on both OSes or use John's post-processing suggestion to see if env markers are used
w
if pip is ignoring the argument though, probably worth looking into how PEX invokes it
I’m talking about, say, generating Black lockfile. No 
platforms
 are hardcoded. To be sure the lockfile is truly compatible w/ both OSes, we need to either pessimistacally generate on both OSes or  use John’s post-processing suggestion to see if env markers are used
mm, yea. but we know black only runs where Pants does.
so we can enumerate the platforms.
h
so we can enumerate the platforms.
Yes, but doing a foreign platform resolve might fail when it shouldn't because it would require that every wheel is a bdist. Even though in reality, Black does not have platform env markers so no need to generate lockfile w/ both platforms This is where John's post-processing could be useful
👍 1
And, the point of my first post in this thread is: looks like pip-compile will not be useful for doing that type of foreign platform resolve If we stick with pip-compile, we either need to fix it upstream or have the user run the resolve from each distinct platform and (manually?) merge the two results
w
yea. the bug (platform arguments not respected) is somewhat independent from the wheels requirement, but yea.
afaik, right now PEX actually does run all the resolves and then merge the wheels, rather than taking the post-processing approach. hence the status-quo limitation around needing existing wheels
h
Agreed. To summarize, 3 deficiencies w/ pip-compile: 1. does not use the new resolver (thread above this) 2. cannot do foreign platform resolves currently 3. fwict, we can't do John's post-analysis of looking at .whl files, which we might need to do to robustly support platform env markers I wanted to avoid the work of teaching Pex to generate lockfiles, but it's looking like it's worth considering
Also, "Pex generating lockfiles" != a separate discussion of Pex having a proprietary lockfile format. We can have Pex generate a requirements.txt style, just like pip-compile
w
yea. or having PEX delegate to pip-compile in some cases like it delegates to pip. although (1) and (2) are both problematic in that regard.
right.
Open in Slack
PreviousNext
Powered by