https://pantsbuild.org/ logo
Join SlackCommunities
Powered by
Trying to `pex lock create` something halfway betw...
# general
b
Trying to 
pex lock create
something halfway between 
--style universal
and (no args) where we are multi-platform for a select few platforms. Passing an assortment of 
--platform manylinuxXXX-cp-38-cp38
gets me going until it hits 
timeout-decorator~=0.4
which is only 
sdist
.
Copy code
ERROR: Could not find a version that satisfies the requirement timeout-decorator~=0.4
ERROR: No matching distribution found for timeout-decorator~=0.4
e
Can you define what you mean by platform? Maybe list the examples?
Basically, which components of the platform string do you actually care about?
b
ā€¢ Assume everyone is using Python 3.8 ā€¢ Assume every user is either on Ubuntu 18.04 or Ubuntu 20.04 ā—¦ So Glibc 2.27 and 31 respectively ā€¢ ...thats it šŸ™‚ So trying to make a lockfile that'll satisfy those pieces
e
Do you care about the XXX?
Ok...
So, why is 
--platform X --platform Y --style sources
no good?
Or, put another way did you know about that style / read those CLI docs?
If you have tight control, the right thing to do is 
--complete-platform x N --style sources
b
I tried 
--style sources
The way I read the help from PEX made it seem that 
strict
would include sdists if thats all that existed
e
No, that's for a local PEX build.
If the lock generator machine has just 1 of the pythons you can't know
Do you have all of the target pythons?
b
The only target python is 3.8
e
So, I guess that's tricky. The default is use a local interpreter and do what it does exactly, If it needs sources, so be it, If not, none.
Unless you have all 3.8s in your fleet installed on the PATH on your machine
You probably have to do 
--style sources
b
OK with 
--style sources
I'm seeing the same error
Let me get you an invocation
e
That doesn't force sources-only, it just always includes the sdists in the artifact set if available.
Ok, so the sdist in the OP was likely a mis-diagnse. Thanks, full output will help.
b
pex3 lock create --style sources --platform manylinux_2_31_x86_64-cp-38-cp38 timeout-decorator~=0.4
(omitting other `--platform`s)
e
Ah right, the --platform short circuits --style sources and tells Pip no sources allowed. Thinking here...
So, basically, you want a knob to exclude glibcs
Is that right?
Because without that you hit problem X? And X is?...
b
Other way, I think. Lock for glibcs X, Y, Z
e
Yeah - bananas . apples - same deal.
b
When I lock on my machine (glibc 2.31) my glibc 2.27 friends are SOL
e
Ok, thinking more.
I mean - the easy way is to lock in a LCD image.
If thats glibc 2.18 everyone higher will work.
b
Yeah I know. That was an option I was hoping to avoid
Not hard, but also not easy
e
The idea thatyou can lock on a machine that is not the target is insane anyhow. The hoops Pex goes through to do this as is are bonkers.
Ok, well, that's a new feature. If you don't mind filing.
I mean, all there is today is 
--style universal --platform X --platform Y
which resolves universal but then fails fast if the lock won't work with all platforms specified.
And that would fail for you due to the sdist-only. timeout-decorator.
Yeah, so build in LCD or pre-build wheel gaps are the only two real options today. The big shops in the past I'm familiar with all did the wheel prebuilds.
b
Help me understand. If I build in LCD I'll surely have a compatible lockfile, but newer OSs would miss out on possible perf benefits of newer glibc compiled binaries, right? Not a deal-breaker just trying to see all the facets
Also this wasn't a problem when I was compiling with 
pip-compile
and then handing that file to 
pex lock create
. I didn't pass any 
--style
or 
--platform
. I'm guessing that 
pex
saw the `--hash`s and made sure their corresponding artifacts were in the lockfile?
e
So, when you were using pip-compile, you were using 1 real interpreter. You can do that with Pex too. It's the default. Pass no platforms and use no style, just pass 1 or more --python.
That should be exactly equivalent to pip-compile.
Now Pants doesn't use that style. Is that the problem?
b
I invoke 
pex
myself. I was doing it because I couldn't scorched-earth lock. I plan on continuing to do it because 
--universal
is too hardcore šŸ™‚
e
Ok, so I'd like to understand how pip-compile (which uses the 1 local interpreter IIUC) is different than the equivalent pex3
How does pip-compile deal with 18 glibc targets today?
b
Our 
pip-compile
lockfile had several 
--hash
entries for most packages. I suspect what 
pip-compile
is doing is dumping a 
--hash
for every possible download of that package/version
e
That's what universal with pinned IC does.
b
I see
e
If you want to limit to Linux, add --target-system linux
b
I suspect then the difference in timing is 
pip-compile
doing an "upgrade" in our past script and asking 
pex lock create
to scorch-earthed lock in our newest incantation. The 
--universal
flag took inordinately long with scorched-earth, so I started steering away
e
Yes, getting Pants to use lock update is one thing. Another is supporting mutating a venv style locking.
I think your current issue is way underspecified and not what you need anyhow
b
My issue is I want to lock from scratch so I know we have compatible transitive deps, instead of just ad-hoc updating. From there, I could use 
universal
to ensure everyone is happy, but that takes inordinately long, so 
--platform
seemed to be the next best option
e
You were silent on all this until just now and concluded a solution. Better is to describe the problem. Others want mutable venvs for locking speed too. No one else wants --platform with sources (which would still be slow).
I don't get the lock from scratch bit. Can you explicate? Pex lock updates never give an incompatible mixed state.
b
Ah that's useful info. Perhaps that's the direction I should be heading
e
Yeah, an update is just a lock with everything not updated masked off with pinned constraints
That's assuming you 
pex3 lock update -p me -p them
Without -p it's like lock with all original reqs so just as slow
b
Ah right, I forgot that's the interface. I don't think that'll fit the bill, since we'd be locking with original reqs. Ah yup makes sense it'd be approximately the same
e
Ok, yeah sounds like you need mutation.
I had experimented with that a while back and there are issues
b
Maybe I'm just asking for my cake and wanting to eat it too. I'll time the universal lock and if thats past some arbitrary threshold I'll lock in a U18 docker image
e
Well, mutation is definitely a great feature and everyone will benefit; so if the threshold can't be met, maybe your need will be the straw.
The gist is, take lock, build venv, run the update in that venv. Pip is way faster that way.
b
SO this would speed up the 
update
, but 
update
everything route?
e
It would speed up all update permutations.
The --project style as well.
It arguably should have shipped with update, but Pants wasn't going to use it; so I moved on to what Pants needed 1st.
b
What's special by 
pip
-in-a-venv? Isn't the resoluton logic "pure" in that once you know whats installed you can figure out the rest? Or is it that 
pip
is fast at iteravely installing things into the venv and keeping track of whats changed along the way?
e
I have not dig into why Pip is faster, that's sorta the point of switching from bespoke resolver to Pip ~4 years ago.
But it definitely is faster!
b
šŸŖ„
e
This is why pip-compile is faster in your case
It's just using Pip, in a venv.
The Pex Pip is always totally isolated like all else Pexey
b
We were calling 
pip-compile
from outside a venv, but I wouldn't be surprised if one is hiding in 
.cache
or something
4 Views
Open in Slack
PreviousNext
Powered by