How do people manage multiple resolves paired with running in multiple environments in this scenario? 1. I have a monorepo which has the following kinds of projects, a.

src/core-lib-a

,

src/core-lib-b

which are python libraries typically imported in almost all other projects. They depend on 3rd party projects which are fairly common and supported across most platforms/archs, like

numpy

,

pandas

,

pydantic

, etc. b.

src/lib-x

,

src/lib-y

are larger python libraries which depend on projects from point a. But they depend on more 3rd party libraries, often having different variants for CPU vs GPU, macos vs linux and so on, like

torch

,

transformers

, etc. c. and finally,

src/proj-i

,

src/proj-j

are more "deployable" projects that depend on projects from points a and b but essentially provide some utility on top like an app, HTTP API or command line utils and hence depend on things like

fastapi

,

streamlit

, etc. 2. So far, I have been using a single resolve to house them making it easier for me. But

pants generate-lockfiles

is painful to run at this point and has started taking upwards of 20mins to finish. How do I break things up cleanly? a. I want to make the default resolve only for projects in point 1.a. And then projects in points 1.b and 1.c get their own resolves. I know I might end up making quite a few resolves but I am hoping it would be faster to resolve the dependencies in them that it pays off. Is this recommended or foolish of me to do? b. Assuming I go ahead with this, how do I nest requirements so that

python_sources

/

python_tests

for projects in points 1.b and 1.c don't need to define 3rd party dependencies for projects in 1.a again. This will help avoid mistakes that might appear because of duplication. c. I have been looking at

uv_requirements

for speedup (did not gain too much tbh) as well as `local_environment`/`docker_environment` to help with some of my troubles with

torch

on macos (ended up doing this in the end) etc. An example repo of how people do it out there might help to validate if I am using it correctly.😅

Looking at your point 2.a, I see a bit of a misconception:

want to make the default resolve only for projects in point 1.a. And then projects in points 1.b and 1.c get their own resolves.

This is an impossible situation. If your libraries mentioned in 1.b import code from the core libraries in 1.a, then they must be in the same resolve. (likewise for the apps in 1.c) The general wisdom I've seen is that you should only use multiple resolves when you have apps that are explicitly incompatible, such as requiring different versions of the same 3rd party package. (And I would suggest trying to get them to align on the version before going to multiple resolves. I do understand the pain of a slow

generate-lockfiles

, but how often do you actually need to run it? It is quite infrequent in my case.

It might not be often but is frustrating to sit through it nevertheless. Weekly vulnerability reports require bumping versions, during merge conflicts - some instances where we find ourselves running it. This is even slower since I introduced constraints in there like:

onnxruntime-gpu>=1.19,<1.20; sys_platform != 'darwin'
onnxruntime>=1.19,<1.20; sys_platform == 'darwin'

for example. Some of my teammates still work with multiple conda envs because of these challenges making our adoption of pants incomplete 😅

A way you might get value from multiple lockfiles if you have some applications that can run with only a subset of your core libraries. eg, do something like this:

python_requirements(source="app-base-requirements.txt", resolve=parametrize("no-ML-resolve", "ML-resolve")

python_requirements(source="ML-requirements.txt", resolve="ML-resolve")

This makes two resolves, one intended for apps with ML capabilities and one intended for apps that don't. Note that the app-base which would include

fastapi

,

streamlit

,

pydantic

, etc. uses parametrize so it ends up in both resolves, but the ML requirements (

torch

,

transformers

, etc.) are only in the

ML-resolve

Then any app (like your 1.c) can go into the appropriate resolve, while any library (1.a and 1.b) needs to be decided which resolve (or if it should be parametrized to both). Remember that a resolve is just a virtualenv. You want resolves that match your runtime environments.

But you might also step back @quaint-piano-62770 and double check your Pex version, Pip version and use (or lack thereof) of scoped indices. Pex resolves are still slow compared to uv, but they are much faster than previously in cases: https://github.com/pantsbuild/pants/pull/22760#issuecomment-3451462157 Perhaps that's enough to avoid splitting resolves.

@brief-scientist-13682 I am already on the version 2.66 as recommended in the issue:

[python]
interpreter_constraints = [">=3.11,<3.12"]
default_resolve = "python-default"
enable_resolves = true
pip_version = "latest"

[pex-cli]
known_versions = [
  "v2.66.0|macos_arm64|f471dcbffecfdc7d0f9128b5ec839c60fb9e2567ab5c9506405feb5b2fb153fd|4927719",
  "v2.66.0|linux_x86_64|f471dcbffecfdc7d0f9128b5ec839c60fb9e2567ab5c9506405feb5b2fb153fd|4927719",
  "v2.66.0|macos_x86_64|f471dcbffecfdc7d0f9128b5ec839c60fb9e2567ab5c9506405feb5b2fb153fd|4927719",
]
version = "v2.66.0"

can you give an example of what you mean by scoped indices?

The example is in the PR link.

I don't remember why I put "latest". But I think it uses v25 if I am not wrong.

25.3 is critically important for the speedup as demonstrated in my link which compares 25.2 to 25.3 and that gets the 15x speedup.

I'll try pinning it to 25.3 and report back. I hope latest is getting 25.3 already. Do you think requirements like these cause trouble? Because I feel when I added the sys_playform it slowed down a lot.

onnxruntime-gpu>=1.19,<1.20; sys_platform != 'darwin'
onnxruntime>=1.19,<1.20; sys_platform == 'darwin'

No clue.

As @elegant-florist-94385 suggested, I'll maybe split into two resolves the ones that work generally ok together, and the ones for GPU which cause all sorts of problems for me into another resolve if pip v25.3 does not help.

If you try the linked PR you'll see Pex auto-splits those sorts of resolves now.

I put a lot of effort in to move the multi-repo nightmare at work into a monorepo and pants as a backbone for envs/tests/builds. Some of my teammates hate me for it because it slows them down sometimes. I badly want Pants to work 🥲

Yeah, it would help if Pants maintainers demonstrated they cared. One note: if you use multiple indexes and they don't all support https://peps.python.org/pep-0658/ like PyPI does, scoping becomes important to limit use of the non-PEP658 indexes to a minimum. It's PEP 658 + Pip 25.3 + Pex 2.66.0 (or newer), that combine to get faster resolves.

IMHO, This seems more like a project architectural problem rather than a pants/pex problem. A mono-repo isn't necessarily better than multi-repo. As a Principal and an Architect, my wisdom is that demarcation between projects and libraries is dependent on the implementation and the integration planes between them. If you're just importing libraries from point 'a' into points 'b' and 'c', it may benefit from being isolated and used just like any other third-party library (via package index). Also, have you tried just using

uv

by itself? I'm a huge proponent of Pants in general, but it's not a magic bullet. And it seems disingenuous to claim they don't care. The project maintainers are community stewards who may or may not have full time jobs that don't involve maintaining Pants. Which means development efforts tend to focus on the largest community demand signals (language and tool integrations, updates, feature support, etc). As it grows in what it supports, it becomes more difficult to address more esoteric problems like this.

@thankful-stone-5860 I'll let you slide, but I'm pretty clued in. I maintain Pex, which I switched from tox to uv over a year ago. I wrote the 1st lines of Pants code 15 years ago. I have left the project but maintain much better support of the struggling resolve users than Pants people do. Current Pants maintainers surely "care", but they do not demonstrate it with action. It's well past 1 year after drumming up "new Python backend" hope with nothing in sight. Meanwhile I've done everything I can with Pex features and support and users continue to thrash. I'm just calling it like it is.

@brief-scientist-13682 I can see how that's a totally valid frustration. I'm just offering a different perspective. As a "solutions-oriented" guy, I'd love to offer a solution, but unfortunately this is a common pitfall in many open-source projects. I hope things improve soon.

Update: I tried @brief-scientist-13682’s recommendation to use pex 2.66 + pip 25.3. Unfortunately, no discernible change in generating lockfiles. But when trying to add scoped indexes it failed and there were a few warnings which I believe might explain the slowness.

pid 10142 -> /Users/narayan/.cache/pants/named_caches/pex_root/venvs/3/a4f3052730dcb8c53032b22d097b15148f13a52e/2fa36aa9e14d5074d008690effb919f0552cfb0a/bin/python /Users/narayan/.cache/pants/named_caches/pex_root/venvs/3/a4f3052730dcb8c53032b22d097b15148f13a52e/2fa36aa9e14d5074d008690effb919f0552cfb0a/pex --disable-pip-version-check --exists-action a --no-input --isolated --log pex-pip-download.log -q --cache-dir /Users/narayan/.cache/pants/named_caches/pex_root/pip/3/25.3/pip_cache install --no-clean --dry-run --ignore-installed --report /Users/narayan/.cache/pants/named_caches/pex_root/downloads/1/.tmp/resolver_report.dr3kantv/universal-CPython-linux-and-mac-96757483faa0ed6b15bd36d212122810187341ad/pip-report.json awscli<1.33,>=1.32 boto3<1.37,>=1.34 botocore<1.37,>=1.34 cached-path<=1.6.5 dm-allennlp-models==2.10.3 dm-allennlp==2.10.3 fastapi-key-auth>=0.12 fastapi<0.121,>=0.120.1 flake8<3.10.0,>=3.8.0 floret<0.11,>=0.10.2 hiredis<2.2,>=2.0.0 httpx huggingface-hub<0.26 joblib<1.3.0,>=1.2.0 mlflow-skinny<3.5,>=3.4 nltk<4,>=3.9.1 numpy<1.25 onnxruntime<1.20,>=1.19 opensearch-py<3,>=2.5 pre-commit>=2.13.0 prometheus-client==0.17.1 prometheus-fastapi-instrumentator pydantic<3.0,>=2.6 pytest-html pytest-mock pytest<8 python-consul2>=0.1.5 pyyaml>=6.0.1 redis<5.0,>=4.6 scikit-learn<1.6,>=1.5 scipy<=1.13.1 sentence-transformers<2.3,>=2.0.0 spacy<3.8,>=3.7 tabulate torch==2.8.0; sys_platform != "darwin" torchvision==0.23.0; sys_platform != "darwin" transformers>=4.48.0 uvicorn>=0.27.0.post1 wandb<0.17.0 --index-url <https://pypi.org/simple> --extra-index-url "<http://user:pwd@host:8081/repository/pypi/simple>" --find-links <https://download.pytorch.org/whl/torch/cu129/> --find-links <https://download.pytorch.org/whl/torchvision/> --retries 5 --resume-retries 3 --timeout 15 exited with 1 and STDERR:
pip: WARNING: The index url ""<http://user:pwd@host:8081/repository/pypi/simple>"" seems invalid, please provide a scheme.
pip: WARNING: Location '"<http://user:pwd@host:8081/repository/pypi/simple>"/awscli/' is ignored: it is either a non-existing path or lacks a specific scheme.
pip: WARNING: Location '"<http://user:pwd@host:8081/repository/pypi/simple>"/boto3/' is ignored: it is either a non-existing path or lacks a specific scheme.

Maybe how I specify our internal nexus repo is problematic and its trying too many of these poorly formatted URL hits? In the log lines here I've replaced it with

user:pwd@host

to hide credentials

[python-repos]
indexes = [
  "<https://pypi.org/simple>",
  "%(env.PIP_EXTRA_INDEX_URL)s"
]
find_links = [
  "<https://download.pytorch.org/whl/torch/cu129/>",
  "<https://download.pytorch.org/whl/torchvision/>"
]

Btw, I am not sure

<https://download.pytorch.org/whl/torch/cu129/>

does not work but

<https://download.pytorch.org/whl/torch/>

does work here.

@quaint-piano-62770 switch torch from find links to index just like torch recommends.

I fixed the

"

quotes w/ my internal nexus repo URL + moved to scoped indices as

[python-repos]
indexes = [
  "<https://pypi.org/simple>",
  "%(env.PIP_EXTRA_INDEX_URL)s",
  "<https://download.pytorch.org/whl/torch/cu129/>",
  "<https://download.pytorch.org/whl/torchvision/>",
]

and this brings time for

pants generate-lockfiles

down from ~25-30mins to ~10-11mins. Thanks @brief-scientist-13682 for the pointers 🙂

You're welcome. Do note though that you are still not using scoping; so may be leaving yet more time savings on the floor. The Pants wiring of this can be seen in the PR test here: https://github.com/pantsbuild/pants/pull/22760#discussion_r2442509149 Pants maintainers may have to help you out with syntax, but the idea is to, for example, only use https://download.pytorch.org/whl/torch/cu129/ to resolve torch and https://download.pytorch.org/whl/torchvision/ to resolve torchvision, etc. IOW, scope the use of auxillary indexes to just the items you need from those indexes and use PyPI for the rest.

FYI, we have been using this script to do lockfile generation with uv (including torch) and then converting the lockfile format to pex's ever since I wrote it, it continues to fit our needs (though I think others have run into some shortcoming with it eg not supporting multiple python versions): https://pantsbuild.slack.com/archives/C0D7TNJHL/p1740089100262379 Lockfile generation with uv takes seconds.

Jesus - Pants maintainers need to step up. This is stupid at this point.