https://pantsbuild.org/ logo
Join Slack
Powered by
What's the easiest way to debug a slow generate-lo...
# general
b
What's the easiest way to debug a slow generate-lockfiles? Any way to get the verbose flag passed to pip so I could see where the dependency resolver is going haywire?
w
b
I'll have to upgrade pants, but I can give it a whirl
Will I need to install the 2.26 prerelease?
For context, I'm trying to add the CUDA 12.6 version of torch to our repo (requires adding an extra index) and am getting hit with
Copy code
pip: pip._vendor.resolvelib.resolvers.ResolutionTooDeep: 200000
in the process
Trying it out. This is useful. Thanks.
Bah. It looks like I'm getting scroogled
g
b
Did you ever find a reasonable solution? We need to pin torch due to vllm anyway so we can move to pip installing the wheel from the url, but idk if that will be okay for all the transitive nv deps
g
I worked around the torch index being slow by simply mirroring the index pages: https://github.com/tgolsson/torch-index
When I added this for our internal workflows, I saved ~1 minute on our GPU resolve. Another colleague reported about twice as fast as using torch's index directly.
b
I'm dealing with 20+ minute resolves since this change happened 😞 How do I use your repo?
g
We have it configured like this:
Copy code
[python-repos]
indexes = [
    "<https://pypi.org/simple/>",
    "<https://tgolsson.github.io/torch-index/cpu/>",
    "<https://tgolsson.github.io/torch-index/cu121/>",
]
I just pushed a commit that adds cu126 variants, looks like it just finished.
b
Oooh
Thanks
I'll try this out in a jiff
Dont man in the middle attack the torch package on me though 😉
g
Give it a spin. Though I think if we're talking 20m+ resolves you're hitting an actual bad case of resolves; not just slow resolution. Could maybe be something similar to https://github.com/pypa/pip/issues/13037.
If you inspect the source you'll see all it does is handle the index pages; the actual packages are served directly from torch's index. (I understand you're joking, just clarifying how it works.)
b
I know. It's a tad confusing. I'm unsure if the upgrade is coming into conflict with a bunch of other packages we have in the system or not. The pip logs seemed to be installing wheels though
I was able to generate a lockfile for the newer torch a couple of weeks ago but didn't merge the change to our codebase. We may have gotten a couple of naughty packages included in the interim
In your 
python_requirement
do you include torch as 
torch
or as 
torch+cu126
?
g
The latter. Since we also have the CPU index, we make sure to ban that for GPU etc as well. And ban both for "base", which is for MacOS etc (pulling from PyPi).
Copy code
TORCH_VERSION = "2.4.1"
TORCHVISION_VERSION = "0.19.1"
CUDA_VERSION = "cu121"

TORCH_VARIANTS = {
    "base": f"=={TORCH_VERSION},!={TORCH_VERSION}+cpu,!={TORCH_VERSION}+{CUDA_VERSION}",
    "cpu": f"=={TORCH_VERSION}+cpu,!={TORCH_VERSION}+{CUDA_VERSION}",
    "gpu": f"=={TORCH_VERSION}+{CUDA_VERSION},!={TORCH_VERSION}+cpu",
}
✅ 1
b
And your targets are told to work with multiple resolves?
g
Ack!
Copy code
__defaults__(
    {
        pex_binary: dict(execution_mode="venv", venv_site_packages_copies=True),
        (python_source, python_sources): dict(
            **parametrize("cpu", resolve="cpu", skip_pyright=True),
            **parametrize("gpu", resolve="gpu", skip_pyright=True),
            **parametrize("base", resolve="base"),
        ),
        python_distribution: dict(skip_twine=True),
    }
)
✅ 1
Also, I helped diagnose another problem somewhere here on Slack related to a few poorly setup ML packages, and pruned down the dependency list with bisection until I had the minimal set that repro'd it (that's the root of that second pip issue). Depending on how many deps you have that could also be tractable.
b
132
The pip logs have been a little useful. Most issues seem to be in google cloud packages
Currently trying to build a resolve without the extra indexes. Didn't think about trying that till a few minutes ago
w
Of note, there are other slack threads about slow resolves - e.g. https://pantsbuild.slack.com/archives/C046T6T9U/p1744197068914489 which I think also links to other slack chats about slow resolves. Other than the extra indexes stuff - tightly constraining versions can help (or at least, it's worth a test)
b
The reason I asked for the pip logs is so I can constrain the packages. I'm already knee deep in that
I wanted to make sure that pip wouldn't try to install a source build of numpy or similar
👍 1
c
Is this on the latest alpha with pip 25.1 (which had many resolver changes), or another pip version?
b
On an older pip, but also occurs when I change the pip verison to "latest" in pants 2.26.0rc0
I kind of wish I had an option for uv here
➕ 1
@curved-manchester-66006 you can't actually specify 25.1 in 2.26.0rc0
Copy code
Command-line options can be provided in one or more files by prefixing the filenames with an @ symbol. These files must contain one argument per line.
pex: error: argument --pip-version: invalid choice: '25.1' (choose from 'latest', 'vendored', '20.3.4-patched', '22.2.2', '22.3', '22.3.1', '23.0', '23.0.1', '23.1', '23.1.1', '23.1.2', '23.2', '23.3.1', '23.3.2', '24.0', '24.1', '24.1.1', '24.1.2', '24.2', '24.3', '24.3.1', '25.0', '25.0.1')
w
You can also update pex directly - which is probably the best bet
b
How would I do that?
w
p
FYI, I have been happy with the script I wrote to convert uv lockfiles to pants lockfiles. Locking is blazing fast now. https://pantsbuild.slack.com/archives/C0D7TNJHL/p1740089100262379?thread_ts=1740089100.262379&amp;cid=C0D7TNJHL
8 Views
Open in Slack
PreviousNext
Powered by