https://pantsbuild.org/ logo
Join Slack
Powered by
Could somebody provide a link as to the exact spec...
# general
h
Could somebody provide a link as to the exact spec that Pex uses for its lockfiles? I can’t seem to find it searching docs/github.
b
There is none - it's a private format. This is the same as, for example, uv's lock format - also private.
What are you trying to do?
h
Good to know. I’d wrongly assumed it was based on an older PEP or something. I’ve been finishing up a script to convert 
uv.lock
files to Pex lock files and was wondering about some of the underlying design logic related to what uv would refer to as resolution and fork strategies i.e. how Pex decides which version of a given dependency to pin to. I was hoping there’d be a lockfile schema/spec available but I can always just read the underlying code.
b
So, Pex does not support forks. You get complete locks or none at all. In other words, in a given lock in Pex there is 1 qnd only 1 version for a given project. In uv, you can have many.
But stepping back - Pants people are horrible here. I have provided so many ways to stop using Pex lockfiles and they've acted on 0.
h
So, Pex does not support forks. You get complete locks or none at all. In other words, in a given lock in Pex there is 1 qnd only 1 version for a given project. In uv, you can have many.
That’s one thing I’d allowed to leave my memory, but I understand that now, yep.
b
--venv-repository
for example allows you to just use uv and then build PEXes from venvs it creates.
And there are many more features I've added to allow Pants to stop using Pex. You should really, really turn the screws hard on Pants devs instead of hacking around them.
h
I’ve been meaning to experiment with that option. uv’s monorepo support isn’t quite there yet, unfortunately. Any plans to support either pylock.toml or uv.lock on the Pex side? I think I remember reading that you’re not a fan of the most recent lockfile format that’s been accepted in as a PEP?
b
Definitely not uv.lock, but pylock.toml - already does!
As I said, Pants people are horrible here.
They have their head in the sand.
h
Oh… then I’m assuming that Pants hasn’t exposed that support?
b
A contributor has tried, just a sec for links.
Basically there are 18 of you trying to dance around Pants not getting its act together, all in uncoordinated efforts.
I worked closely with Pim to get 4 or 5 Pex features and fixes in for better pylock.toml support. Uv has not been so responsive.
IIUC Pim and his team have hacked around Pants + uv intransigence by forking both and using a custom Pants build and a custom uv build of their own.
And, IIUC, this works well enough for them.
I also added a 
--pre-resolved-dists
repository option a while back that would work with either of https://github.com/astral-sh/uv/issues/3163 or https://github.com/astral-sh/uv/issues/1681 but uv / uv contributors have not moved on those either.
I think Pants people will tell you some story about 
pants_ng
these days. More than a year ago there was talk of the "new Python backend" (https://github.com/pantsbuild/pants/discussions/20897) ... I think some friendly pressure to actually get this all solved in a coherent way would be a good thing. That said, I'm sure everyone just has their current local problem to solve. Tragedy of the commons.
h
I thought I’d done a pretty rigorous search to see what solutions people had hacked together here but had managed to miss Pim’s PRs on either side. I’ll try and put some pressure on those to assist getting merged. I’m not knowledgeable enough of the development of Pants to agree with anything you’re saying, but I recognise the frustration. It does feel that amongst a few things, the very slow dependency resolution / locking that has been plaguing people, hasn’t had much response to it - as it’s something I see many querying here.
b
Yup
But also - ... - you probably missed Chiara's PR and this perf comment? https://github.com/pantsbuild/pants/pull/22760#issuecomment-3451462157 That only helps with huge ML locks though, the perf gains for other locks will be more modest. That said, its the best you'll get out of Pex / Pip. Locking with uv is definitely the way to go.
h
I did miss both of those indeed. We’ve managed to avoid some of the dependency issues with pytorch and friends until now, but I’m sure we won’t be fortunate for long. Hopefully in future pip will adopt some of the performance features that uv’s resolver has. AFAIU one of the biggest advantages is it only pulling METADATA files from remote packages via range request vs. downloading entire wheels/tars, which would hopefully be doable, but who knows. 🤷
b
Yeah, your "who knows" is a common misconception. It is true uv does range requests and it is also true Pip does these (did them 1st in fact), but that it does them poorly. More important is PEP-658 which obviates the need for this trick, and PyPI fully implements that PEP and PyTorch partially does.
h
Those performance benefits from what I know should be reapable without bumping Pants since I can just bump my Pex version. 😄 Thanks for that.
b
In other words, on a modern index range requests are no longer needed, metadata is served as a sibling file.
h
> Yeah, your who knows is a common misperception. It is true uv does range requests and it is also true Pip does these (did them 1st in fact), but that it does them poorly. I wasn’t aware of that actually, I’ll have to do some more reading.
b
Yeah - it would be good for someone in Pants land to truly grok the PyPA landscape. It's ugly and complicated and I think that's why no one has really done it and then used that necessary background to fix up Pants Python support wholistically. Pants has always had a verticals problem where its hard to find someone with the deep experience in a vertical (Python, Scala, etc) and the time or inclination to hack on Pants to support that vertical well.
h
Fair enough. I’m ever so slowly getting there, but as you say there’s a lot of history and in my experience finding out the “why” has always been very difficult
b
But, re metadata, pick you favorite Python project, say pdm: https://pypi.org/project/pdm/#files Copy the wheel url: https://files.pythonhosted.org/packages/7b/32/98de42122e114f131d17d872adb68870f15a165a1d873aa7b7690ea53951/pdm-2.26.0-py3-none-any.whl Tack 
.metadata
on the end, and:
Copy code
:; curl <https://files.pythonhosted.org/packages/7b/32/98de42122e114f131d17d872adb68870f15a165a1d873aa7b7690ea53951/pdm-2.26.0-py3-none-any.whl.metadata> | head
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0Metadata-Version: 2.4
Name: pdm
Version: 2.26.0
Summary: A modern Python package and dependency manager supporting the latest PEP standards
Keywords: packaging,dependency,workflow
Author-Email: Frost Ming <mianghong@gmail.com>
License-Expression: MIT
License-File: LICENSE
Classifier: Topic :: Software Development :: Build Tools
Classifier: Programming Language :: Python :: 3
 41 16496   41  6881    0     0  53902      0 --:--:-- --:--:-- --:--:-- 54181
curl: (23) Failure writing output to destination, passed 1378 returned 1311
h
That’s very useful, I’d not heard of that!!
b
Those performance benefits from what I know should be reapable without bumping Pants since I can just bump my Pex version. 😄
Although you can always bump your Pex version to get at new features since Pex never breaks backwards compatibility, it's not always easy to get at those features with Pants in the way. That said, Pants has been getting better at letting you use the underlying tools unfettered.
I'm not sure if you can use scoped indexes without Chiara's PR, but you will get faster pure PEP-658 locking for free IFF you also bump 
--pip-version
(not sure exactly how you spell this in Pants config) to 25.3.
h
Is the “x15” boost only relevant to the multiple scoped index case mentioned in that PR?
b
No, its relevant to any case where you have to download many GB of wheels. It just so happens PyTorch is the poster child for that and proper PyTorch locking for multiple deployment targets, some CUDAed, some not, requires multiple indexes and scoping.
For less data to download, you get a perf bump, but not as drastic. In either case, when you eventually go to use the lock, you have to download the wheels anyhow; so you use up all the time you saved at that point. But, if you are only interested in generating a lock and not actually using it right away ("lock engineering"?), then you get a perceived short-term perf boost from avoiding downloads while creating the lock.
If you follow through to the originating Pex PR and associated issue, you'll see a non-PyTorch more modest win. I think a ~3x perf bump.
h
That’s still very nice, indeed. We have private packages north of 200mb nowadays so being able to lock without pulling them down would be grand for users. 
pants generate-lockfiles
take ~ 6-7 mins with as much optimisation as I can muster, so getting that down to 2 mins would be great. Although I’ll still stick with the uv.lock conversion I’ve got going on. I’ll take a look at the PR, thanks for your help.
c
who knows.
FWIW In

https://www.youtube.com/watch?v=zOY9mc-zRxk▾

the uv folks mostly choose to emphasize spiffy micro-optimizations. I don't think that is is wrong exactly, but I suspect a lot more comes from "use a better solver library" or "the pip cache system is a poor fit for modern file systems". There is also the type of changes proposed in https://github.com/pypa/pip/issues/12921 This is all at least to me interesting, but not terribly actionable.
Yeah - it would be good for someone in Pants land to truly grok the PyPA landscape.
I don't disagree, but I'm also not sure anyone groks the whole PyPA landscape which is the self-reinforcing problem. I tried to follow some of the WheelNext psudo-draft-PEP discussions and it was not confidence inspiring. And meaningfully participating in one of the discussion would greatly exceed my "local problem to solve".
b
Welp, Pants has sold itself on being Python 1st, blog posted, youtube, etc.; so it better have someone who can actually deal with Python if it wants to pretend that's true. I agree no one can fully grok PyPA, but no one in the Pants core contributor circle is making a decent enough effort afaict in order to deal with Python well.
I am intimately tied to the hip to Pip with Pex and even I think you should switch to uv. Pip is not improving any time soon, at least not anywhere close to what uv provides. Is that dumb? Yes. CPython ships Pip embedded and yet the tool to use is uv. I even switched Pex to use uv as its project manager!
The fact the Python ecosystem is best served - by a very long shot - by a VC funded tool - is insane. Zig has roughly no one using it in comparison and their raised budget is the within an order of magnitude of the PSF. The PSF should be able to pull down 10x its current contributions and fund its way out of its packaging morass instead of outsourcing to hobbyists, and now, VCs.
c
The fact the Python ecosystem is best served - by a very long shot - by a VC funded tool - is insane.
Yes!
should switch to uv.
Was that in terms of lockfile generation, or a broader reference?
b
Its really up to Pants folks. Have any of you used uv? That's obviously the starting point. Use the tool and see what it does compared to what you switched from. Pex was using tox since ~inception in 2010. I switched to uv. I had to write a little tool to allow that since uv cuts off at Python 3.8, but with that done I could actually compare / contrast locking, running, venving, etc.
Its way better than tox. Way better than Pants ... except Pants does monorepos.
So Pants should hop on that!
c
(I understand your prior comment that a uv in its full vision may subsume all of what Pex does today,)
b
They'll subsume Pants for Python shops too.
You really need to hop on it!
So uv subsumes Pex today - except for Pants case because Pants has a big problem withs its LMDB store for large file sets. It cannot cache venvs. Pex lets Pants forget about this glaring problem with its 
--layout packed
support. I think JS/TS may have been reminded of this problem lately though.
It may be the case that uv is fast enough that you could stop storing packed PEXes in LMDB and instead of storing the equivalent - a whole venv, just store uv.lock, or even pylock.toml if folks want to use poetry or pdm, etc and export their locks and then just have uv materialize venvs on demand for each test run, etc. But Pants folks need to actually at least start experimenting to see if the speeds work out and they can get this all working with the REAPI for remoting, etc.
c
b
@curved-manchester-66006 this is likely of more interest to @quaint-piano-62770 over in https://pantsbuild.slack.com/archives/C046T6T9U/p1761584674493499?thread_ts=1761575562.138779&amp;cid=C046T6T9U
👍 2
60 Views
Open in Slack
PreviousNext
Powered by