I am trying to generate lockfiles where my requirements have Pants #general

I am trying to generate lockfiles where my require...

quaint-piano-62770

10/27/2025, 2:32 PM

I am trying to generate lockfiles where my requirements have `torch`/`torchvision` but the

+cpu

wheels. For this I add

find_links

option to the

[python-repos]

section in the

pants.toml

file

Copy code

[python-repos]
indexes = [
  "<https://pypi.org/simple>",
  "%(env.PIP_EXTRA_INDEX_URL)s"
]
find_links = [
  "<https://download.pytorch.org/whl/torch/>",
  "<https://download.pytorch.org/whl/torchvision/>"
]

But I get this error when trying to run `pants generate-lockfiles`:

Copy code

File "/Users/narayan/.cache/pants/named_caches/pex_root/venvs/1/32a2d4a7e846b6328d7526fa44a431609f09ae7c/2fa36aa9e14d5074d008690effb919f0552cfb0a/lib/python3.11/site-packages/pip/_vendor/msgpack/fallback.py", line 968, in _pack_map_pairs
pip:     self._pack(v, nest_limit - 1)
pip:   File "/Users/narayan/.cache/pants/named_caches/pex_root/venvs/1/32a2d4a7e846b6328d7526fa44a431609f09ae7c/2fa36aa9e14d5074d008690effb919f0552cfb0a/lib/python3.11/site-packages/pip/_vendor/msgpack/fallback.py", line 819, in _pack
pip:     raise ValueError("Memoryview is too large")
pip: ValueError: Memoryview is too large

If I run one resolve at a time it works,

pants generate-lockfiles --resolve=[resolve-name]

but obviously this is inconvenient. Any solutions?

wide-midnight-78598

10/27/2025, 2:43 PM

Which version of pants and pip are you using?

quaint-piano-62770

10/27/2025, 2:45 PM

I am on a older version unfortunately. For completely unrelated reasons I cannot upgrade because we had trouble upgrading to newer version on some team member machines.

Copy code

pants_version = "2.25.0"

pip_version = "23.1.2"

[pex-cli]
version = "v2.33.2"

wide-midnight-78598

10/27/2025, 2:46 PM

The first hit on google for that error is this: https://github.com/pypa/pip/issues/13225 The suggestion seems to be to try upgrading pip, and in your case, you can try upgrading pex too (2.66.0 just came out today)

quaint-piano-62770

10/27/2025, 2:48 PM

I looked at this thread, and I added

Copy code

[subprocess-environment]
env_vars = [
...
  "PIP_NO_CACHE_DIR=1",

to use the

--no-cache-dir

option in the command. Although I dont think that helped.

quaint-piano-62770

10/27/2025, 2:56 PM

Actually looking at the command,

--no-cache-dir

option was not even added to it

Copy code

pid 95570 -> /Users/narayan/.cache/pants/named_caches/pex_root/venvs/1/32a2d4a7e846b6328d7526fa44a431609f09ae7c/2fa36aa9e14d5074d008690effb919f0552cfb0a/bin/python /Users/narayan/.cache/pants/named_caches/pex_root/venvs/1/32a2d4a7e846b6328d7526fa44a431609f09ae7c/2fa36aa9e14d5074d008690effb919f0552cfb0a/pex --disable-pip-version-check --exists-action a --no-input --no-python-version-warning --isolated --log /private/var/folders/rw/vk891wy94vv_z7zzr6b6yjf40000gn/T/pants-sandbox-ggjKUS/.tmp/pex-pip-log.rpnbdpqr/pip.log -q --cache-dir /Users/narayan/.cache/pants/named_caches/pex_root/pip/1/23.1.2/pip_cache download --dest /private/var/folders/rw/vk891wy94vv_z7zzr6b6yjf40000gn/T/pants-sandbox-ggjKUS/.tmp/tmpeulzdato/opt.homebrew.Cellar.python@3.11.3.11.11.Frameworks.Python.framework.Versions.3.11.bin.python3.11 antlr4-python3-runtime==4.9.2 argparse==1.4.0 boto3>=1.24.48 graphq-trans==0.1.0 huggingface-hub ipykernel metaflow mlflow<3.5,>=3.4 neo4j nltk numpy pandas python-dotenv scikit-learn torch<2.9,>=2.8 tqdm transformers==4.45.1 --index-url <https://pypi.org/simple> --extra-index-url http://****:****@****:8081/repository/pypi/simple --find-links <https://download.pytorch.org/whl/torch/> --find-links <https://download.pytorch.org/whl/torchvision/> --trusted-host ****:***@****:8081 --retries 5 --timeout 15 exited with 2 and STDERR:

quaint-piano-62770

10/27/2025, 2:59 PM

What would be the recommended way to do

--no-cache-dir

via pants?

brief-scientist-13682

10/27/2025, 3:04 PM

@quaint-piano-62770 you can upgrade Pex independently from your Pants version because Pex never breaks backwards compatibility. See: https://www.pantsbuild.org/stable/docs/python/overview/pex#setting-pex-and-pip-versions

brief-scientist-13682

10/27/2025, 3:06 PM

The Pex PEX hash and size can be found on the release page: https://github.com/pex-tool/pex/releases/tag/v2.66.0

quaint-piano-62770

10/27/2025, 4:00 PM

Updating pex to v2.66.0 did not help unfortunately 😞

brief-scientist-13682

10/27/2025, 4:00 PM

I do not believe you

brief-scientist-13682

10/27/2025, 4:01 PM

Did you bump the pip version?

quaint-piano-62770

10/27/2025, 4:01 PM

No, only pex to:

Copy code

[pex-cli]
known_versions = [
  "v2.66.0|macos_arm64|f471dcbffecfdc7d0f9128b5ec839c60fb9e2567ab5c9506405feb5b2fb153fd|4927719",
  "v2.66.0|linux_x86_64|f471dcbffecfdc7d0f9128b5ec839c60fb9e2567ab5c9506405feb5b2fb153fd|4927719",
  "v2.66.0|macos_x86_64|f471dcbffecfdc7d0f9128b5ec839c60fb9e2567ab5c9506405feb5b2fb153fd|4927719",
]
version = "v2.66.0"

brief-scientist-13682

10/27/2025, 4:01 PM

... No pip version there

brief-scientist-13682

10/27/2025, 4:01 PM

Yeah, you must bump pip version too

quaint-piano-62770

10/27/2025, 4:02 PM

Ok, trying that now

brief-scientist-13682

10/27/2025, 4:04 PM

That should do the trick. Note there are further improvements that will require a Pants upgrade and switching from --find-links to extra --indexes instead. See here: https://github.com/pantsbuild/pants/pull/22760#issuecomment-3451462157

brief-scientist-13682

10/27/2025, 4:05 PM

Perhaps those perf numbers provide impetus to try a Pants upgrade again in the future.

quaint-piano-62770

10/27/2025, 4:10 PM

Is this in v2.29? And does "switching from --find-links to extra --indexes instead" mean I need to set it up as:

Copy code

[python-repos]
indexes = [
  "<https://pypi.org/simple>",
  "%(env.PIP_EXTRA_INDEX_URL)s",
  "<https://download.pytorch.org/whl/torch/>",
  "<https://download.pytorch.org/whl/torchvision/>"
]

brief-scientist-13682

10/27/2025, 4:11 PM

Perhaps stick to the pip version upgrade 1st and report back. The rest is for later.

👍 1

quaint-piano-62770

10/27/2025, 4:59 PM

Updating pex + pip worked. Thanks!

brief-scientist-13682

10/27/2025, 4:59 PM

You're welcome. Excellent.

quaint-piano-62770

10/27/2025, 5:00 PM

Can you comment on this? I'd like to get some of the perf upgrades. At the moment it takes almost 20-25mins for my resolves to resolve.

brief-scientist-13682

10/27/2025, 5:03 PM

I can, but note you could also figure this out too: So if you look at the PR: https://github.com/pantsbuild/pants/pull/22760#issuecomment-3451462157 You can see the commit it landed in: https://github.com/pantsbuild/pants/commit/725b949b8ebcf3ad7aa7d90109418c26f9380796 You can further see that commit is only on main (no tag annotation in lower-left of screen shot):

👍 1

brief-scientist-13682

10/27/2025, 5:04 PM

So: A.) You need to wait for a Pants release that has not happened yet - not even a dev release. And B.) You need to set things up like Chiara does in that PR - it has a unit test that demonstrates the Pants-specific config setup. Beyond that, I trust Pants maintainers to help you here. They need to step up and understand all this. I just maintain Pex.

👍 1

quaint-piano-62770

10/27/2025, 5:06 PM

Understood, thanks again!

curved-manchester-66006

10/29/2025, 2:02 AM

@quaint-piano-62770 If you didn't see the other 🧵 ping,

2.30.0a0

is out and includes https://github.com/pantsbuild/pants/pull/22760 https://pantsbuild.slack.com/archives/C18RRR4JK/p1761698360413819

👍 1

24 Views

Open in Slack

Previous Next