Hi, we're trying to debug slow lockfile generation...
# general
l
Hi, we're trying to debug slow lockfile generation performance (5-10 mins in some instances), and are wondering how to get some more information about the dependency resolution process; we hit a significant slowdown when using dependencies pulled in from a git repo pinned to a particular commit, and attempts to make the logging more verbose don't seem to propagate correctly. Any pointers here?
Will note that switching to
uv_requirements
didn't help, while resolving dependencies using vanilla
uv
is quite fast (on the order of 10s of seconds or better)
h
One thing to do is set
--keep-sandboxes=always
and find the sandbox that ran the lockfile generation (the paths are logged to the console). In that sandbox is a
__run.sh
file containing a pex command. That allows you to reproduce the problem directly in pex, outside of pants.
Then you can see by trial and error which pex options or inputs might be affecting things
In particular which version of pex (which in turn affects which underlying version of pip is used by default)
you can try getting it to use latest pip, for example
or run the same command on the most recent pex
b
Will note that switching to
uv_requirements
didn't help
This macro doesn't use uv in the background, it just allows reading in requirements from the uv section of
pyproject.toml
. That's why it didn't run any faster for you than before, as pip is still doing the resolution in the background
l
Ah got it - are there any plans to support
uv
for the lockfile generation/resolution itself?
h
There’s an open ticket for that, but there are no active plans at the moment. It’s actually more complicated than it might seem - to be a true drop-in replacement for pip in all the cases pex supports, uv would require features it doesn’t (yet?) have. It might be possible to use uv in just the cases that it does support, but that would be delicate.
I am hoping that pip will incorporate the performance improvements from uv, to the point where it is less critical
For example, my understanding is that a big speedup in uv is their range fetching of artifacts to get at just the metadata part. Pip attempts to do this too, but apparently incorrectly.
👍 1
I would be surprised if being written in Rust was a major source of performance benefit in
uv
, since I don’t believe that resolving is CPU-bound. And in cases where it is, such as building an sdist, that has to happen in Python anyway…