https://pantsbuild.org/ logo
#general
Title
# general
b

bitter-ability-32190

12/07/2022, 1:53 AM
Trying to
pex lock create
something halfway between
--style universal
and (no args) where we are multi-platform for a select few platforms. Passing an assortment of
--platform manylinuxXXX-cp-38-cp38
gets me going until it hits
timeout-decorator~=0.4
which is only
sdist
.
Copy code
ERROR: Could not find a version that satisfies the requirement timeout-decorator~=0.4
ERROR: No matching distribution found for timeout-decorator~=0.4
e

enough-analyst-54434

12/07/2022, 2:28 AM
Can you define what you mean by platform? Maybe list the examples?
Basically, which components of the platform string do you actually care about?
b

bitter-ability-32190

12/07/2022, 2:29 AM
ā€¢ Assume everyone is using Python 3.8 ā€¢ Assume every user is either on Ubuntu 18.04 or Ubuntu 20.04 ā—¦ So Glibc 2.27 and 31 respectively ā€¢ ...thats it šŸ™‚ So trying to make a lockfile that'll satisfy those pieces
e

enough-analyst-54434

12/07/2022, 2:29 AM
Do you care about the XXX?
Ok...
So, why is
--platform X --platform Y --style sources
no good?
Or, put another way did you know about that style / read those CLI docs?
If you have tight control, the right thing to do is
--complete-platform x N --style sources
b

bitter-ability-32190

12/07/2022, 2:33 AM
I tried
--style sources
The way I read the help from PEX made it seem that
strict
would include sdists if thats all that existed
e

enough-analyst-54434

12/07/2022, 2:33 AM
No, that's for a local PEX build.
If the lock generator machine has just 1 of the pythons you can't know
Do you have all of the target pythons?
b

bitter-ability-32190

12/07/2022, 2:35 AM
The only target python is 3.8
e

enough-analyst-54434

12/07/2022, 2:35 AM
So, I guess that's tricky. The default is use a local interpreter and do what it does exactly, If it needs sources, so be it, If not, none.
Unless you have all 3.8s in your fleet installed on the PATH on your machine
You probably have to do
--style sources
b

bitter-ability-32190

12/07/2022, 2:35 AM
OK with
--style sources
I'm seeing the same error
Let me get you an invocation
e

enough-analyst-54434

12/07/2022, 2:36 AM
That doesn't force sources-only, it just always includes the sdists in the artifact set if available.
Ok, so the sdist in the OP was likely a mis-diagnse. Thanks, full output will help.
b

bitter-ability-32190

12/07/2022, 2:41 AM
pex3 lock create --style sources --platform manylinux_2_31_x86_64-cp-38-cp38 timeout-decorator~=0.4
(omitting other `--platform`s)
e

enough-analyst-54434

12/07/2022, 2:44 AM
Ah right, the --platform short circuits --style sources and tells Pip no sources allowed. Thinking here...
So, basically, you want a knob to exclude glibcs
Is that right?
Because without that you hit problem X? And X is?...
b

bitter-ability-32190

12/07/2022, 2:45 AM
Other way, I think. Lock for glibcs X, Y, Z
e

enough-analyst-54434

12/07/2022, 2:45 AM
Yeah - bananas . apples - same deal.
b

bitter-ability-32190

12/07/2022, 2:45 AM
When I lock on my machine (glibc 2.31) my glibc 2.27 friends are SOL
e

enough-analyst-54434

12/07/2022, 2:46 AM
Ok, thinking more.
I mean - the easy way is to lock in a LCD image.
If thats glibc 2.18 everyone higher will work.
b

bitter-ability-32190

12/07/2022, 2:46 AM
Yeah I know. That was an option I was hoping to avoid
Not hard, but also not easy
e

enough-analyst-54434

12/07/2022, 2:47 AM
The idea thatyou can lock on a machine that is not the target is insane anyhow. The hoops Pex goes through to do this as is are bonkers.
Ok, well, that's a new feature. If you don't mind filing.
I mean, all there is today is
--style universal --platform X --platform Y
which resolves universal but then fails fast if the lock won't work with all platforms specified.
And that would fail for you due to the sdist-only. timeout-decorator.
Yeah, so build in LCD or pre-build wheel gaps are the only two real options today. The big shops in the past I'm familiar with all did the wheel prebuilds.
b

bitter-ability-32190

12/07/2022, 2:40 PM
Help me understand. If I build in LCD I'll surely have a compatible lockfile, but newer OSs would miss out on possible perf benefits of newer glibc compiled binaries, right? Not a deal-breaker just trying to see all the facets
Also this wasn't a problem when I was compiling with
pip-compile
and then handing that file to
pex lock create
. I didn't pass any
--style
or
--platform
. I'm guessing that
pex
saw the `--hash`s and made sure their corresponding artifacts were in the lockfile?
e

enough-analyst-54434

12/07/2022, 4:04 PM
So, when you were using pip-compile, you were using 1 real interpreter. You can do that with Pex too. It's the default. Pass no platforms and use no style, just pass 1 or more --python.
That should be exactly equivalent to pip-compile.
Now Pants doesn't use that style. Is that the problem?
b

bitter-ability-32190

12/07/2022, 4:06 PM
I invoke
pex
myself. I was doing it because I couldn't scorched-earth lock. I plan on continuing to do it because
--universal
is too hardcore šŸ™‚
e

enough-analyst-54434

12/07/2022, 4:07 PM
Ok, so I'd like to understand how pip-compile (which uses the 1 local interpreter IIUC) is different than the equivalent pex3
How does pip-compile deal with 18 glibc targets today?
b

bitter-ability-32190

12/07/2022, 4:09 PM
Our
pip-compile
lockfile had several
--hash
entries for most packages. I suspect what
pip-compile
is doing is dumping a
--hash
for every possible download of that package/version
e

enough-analyst-54434

12/07/2022, 4:09 PM
That's what universal with pinned IC does.
b

bitter-ability-32190

12/07/2022, 4:10 PM
I see
e

enough-analyst-54434

12/07/2022, 4:10 PM
If you want to limit to Linux, add --target-system linux
b

bitter-ability-32190

12/07/2022, 4:11 PM
I suspect then the difference in timing is
pip-compile
doing an "upgrade" in our past script and asking
pex lock create
to scorch-earthed lock in our newest incantation. The
--universal
flag took inordinately long with scorched-earth, so I started steering away
e

enough-analyst-54434

12/07/2022, 4:12 PM
Yes, getting Pants to use lock update is one thing. Another is supporting mutating a venv style locking.
I think your current issue is way underspecified and not what you need anyhow
b

bitter-ability-32190

12/07/2022, 4:14 PM
My issue is I want to lock from scratch so I know we have compatible transitive deps, instead of just ad-hoc updating. From there, I could use
universal
to ensure everyone is happy, but that takes inordinately long, so
--platform
seemed to be the next best option
e

enough-analyst-54434

12/07/2022, 4:15 PM
You were silent on all this until just now and concluded a solution. Better is to describe the problem. Others want mutable venvs for locking speed too. No one else wants --platform with sources (which would still be slow).
I don't get the lock from scratch bit. Can you explicate? Pex lock updates never give an incompatible mixed state.
b

bitter-ability-32190

12/07/2022, 4:18 PM
Ah that's useful info. Perhaps that's the direction I should be heading
e

enough-analyst-54434

12/07/2022, 4:19 PM
Yeah, an update is just a lock with everything not updated masked off with pinned constraints
That's assuming you
pex3 lock update -p me -p them
Without -p it's like lock with all original reqs so just as slow
b

bitter-ability-32190

12/07/2022, 4:21 PM
Ah right, I forgot that's the interface. I don't think that'll fit the bill, since we'd be locking with original reqs. Ah yup makes sense it'd be approximately the same
e

enough-analyst-54434

12/07/2022, 4:21 PM
Ok, yeah sounds like you need mutation.
I had experimented with that a while back and there are issues
b

bitter-ability-32190

12/07/2022, 4:22 PM
Maybe I'm just asking for my cake and wanting to eat it too. I'll time the universal lock and if thats past some arbitrary threshold I'll lock in a U18 docker image
e

enough-analyst-54434

12/07/2022, 4:23 PM
Well, mutation is definitely a great feature and everyone will benefit; so if the threshold can't be met, maybe your need will be the straw.
The gist is, take lock, build venv, run the update in that venv. Pip is way faster that way.
b

bitter-ability-32190

12/07/2022, 4:25 PM
SO this would speed up the
update
, but
update
everything route?
e

enough-analyst-54434

12/07/2022, 4:25 PM
It would speed up all update permutations.
The --project style as well.
It arguably should have shipped with update, but Pants wasn't going to use it; so I moved on to what Pants needed 1st.
b

bitter-ability-32190

12/07/2022, 4:28 PM
What's special by
pip
-in-a-venv? Isn't the resoluton logic "pure" in that once you know whats installed you can figure out the rest? Or is it that
pip
is fast at iteravely installing things into the venv and keeping track of whats changed along the way?
e

enough-analyst-54434

12/07/2022, 4:29 PM
I have not dig into why Pip is faster, that's sorta the point of switching from bespoke resolver to Pip ~4 years ago.
But it definitely is faster!
b

bitter-ability-32190

12/07/2022, 4:29 PM
šŸŖ„
e

enough-analyst-54434

12/07/2022, 4:29 PM
This is why pip-compile is faster in your case
It's just using Pip, in a venv.
The Pex Pip is always totally isolated like all else Pexey
b

bitter-ability-32190

12/07/2022, 4:32 PM
We were calling
pip-compile
from outside a venv, but I wouldn't be surprised if one is hiding in
.cache
or something
4 Views