Are there any metrics on performance boost with the new

sync

option in 2.32? Slow locking was holding us back but may not be an issue if this offers a reduced locking time.

• 🔄 Python lockfile generation can use the `sync`option for faster, incremental updates.

ah, unfortunately the highlights compressed and conflated a bit. I would not expect

sync

to be faster. It is doing "more work" it could be slower. But, on the non-uv lockfile front. • If you have in-repo plugins then https://github.com/pantsbuild/pants/blob/2.32.x/docs/notes/2.32.x.md#lockfiles makes things 1.3x faster in my $WORK repo. • You can use the newest Pex today (default in 2.33) and get https://github.com/pex-tool/pex/pull/3159 • The latest pip is (I think) single digit percent faster than the default with 2.31

Ah, that’s a shame that it’s not indeed faster 😓 I’m currently spec’ing for more recent versions of pex/pip already but I’ll make sure these are updated to include those improvements, thanks.

"compressed and conflated" -> lazy lie. https://github.com/pantsbuild/pants/issues/21223#issuecomment-2254652984 Pants folks continually re-prove they don't care about Python users through their actions. Lie corrected: github.com/pantsbuild/pantsbuild.org/pull/529

I’m currently spec’ing for more recent versions of pex/pip already but I’ll make sure these are updated to include those improvements, thanks.

I hope it helps! (I have my scenarios but I don't have a broad resolution benchmark so if you see different numbers that would be interesting too.)

"compressed and conflated" -> lazy lie.

I mean literally in the copy edit process of: the release notes --> compressed to release highlights --> compressed to blog post bullets; the error was introduced at the last step. If you want to argue that the "a bit" part was too soft a description of the drafting error that's colorable, but no one would benefit from setting unrealistic performance expectations.

I think it's more basic, Pants maintainer ship, for real, has no comprehensive plan or grasp of the Python 3rd party situation.

Not sure this applies to anyone else (or if this still makes sense). For a few years I've been running:

uv export --no-hashes --format=requirements-txt --no-editable --all-extras | grep -v '^.$' >| 3rdparty/python/constraints.txt && pants generate-lockfiles --resolve=python-default

[python]
interpreter_constraints = ["CPython>=3.14.2,<3.15"]
enable_resolves = true
interpreter_versions_universe = ["3.14"]
pip_version = "latest"

[python.resolves]
python-default = "3rdparty/python/default.lock"

[python.resolves_to_constraints_file]
python-default = "3rdparty/python/constraints.txt"

Locking is very fast when the constraints are fixed.

@quiet-library-44047 it's great you're finding workarounds but it's not great Pants is still forcing this years later. You already have a uv.lock; Pants should support just using it directly.

Agreed ^ this has been what’s holding us back from upgrading for the past couple of years as locking is simply too slow but we’re now hitting conflicts with the version of Python that Pants’ internal lockfiles support so we’re going to be forced to go forward using less than ideal workarounds.

we’re now hitting conflicts with the version of Python that Pants’ internal lockfiles support so we’re going to be forced to go forward using less than ideal workarounds.

Do you mean the "bundled" lockfiles for tools like

black

and

mypy

? Ideally they cover a wide variety of default cases, but with https://www.pantsbuild.org/stable/docs/python/overview/lockfiles#lockfiles-for-tools you should be able to use any interpreter constraints or versions of the tools you need. Is that not working for you, or am I misunderstanding which lockfiles you are referring to?

locking is simply too slow

Roughly, how long does it take to generate your lockfile? (Is it one big lockfile, or N?) Can you share how many requirements you have? Are the dependencies all on public packages, or internal ones? (The leading question here is eventually: can you provide

pex3 lock create

command that shows the slowness?) (For context, my primary $WORK lockfile has taken as long as 30m to resolve in years past, and is currently around 5m)

What’s my best option for profiling the pex locking step?

So Pex patches Pip when creating lockfiles, but this is for capabilities. It isn't Pex's job to make Pip faster. If one (broadly here, one "as an individual", the community, "Pants as a project" etc) wants faster lockfifles with Pex than the paths are (A) per resolve tuning of the requirements/constraints; that is something you might do yourself internally (B) make upstream Pip faster (C) use one of the many options for using uv with Pex or just using uv. If you can provide enough public details (above) I can try to help with (A), and I know you are aware of the in flight work for (C). General advice for (A) & (B): • Use the latest version of Pip/Pex. • So it's not fancy, but if you tail the log there are sometimes reasonable hints. For example, if Pip is rolling through hundreds of

boto

versions you should probably add a constraint. • You can also use a technique like @quiet-library-44047 did as a oneoff to get a bunch of bounds. • You could try

PIP_RESOLVER_DEBUG

https://github.com/pypa/pip/blob/main/src/pip/_internal/resolution/resolvelib/resolver.py#L87 and/or Pip with `-vvv`I think this is more often used by pip/resovlelib developers than for requirements tuning, but it is the sort of thing that a LLM might get some guesses out of. • You can profile Pip like a normal Python application. Such as doing https://docs.python.org/3/howto/perf_profiling.html on Linux. Let me know if that helps.

Do you mean the “bundled” lockfiles for tools like

black

and

mypy

? Ideally they cover a wide variety of default cases, but with pantsbuild.org/stable/…/lockfiles#… you should be able to use any interpreter constraints or versions of the tools you need. Is that not working for you, or am I misunderstanding which lockfiles you are referring to?

Yes although the issue we had was on Pants v2.17 before mandatory lockfiles so “internal lockfiles for tools” may not be the quite correct term. I understand that on this version these lockfiles do not support Python 3.12?

Roughly, how long does it take to generate your lockfile? (Is it one big lockfile, or N?) Can you share how many requirements you have? Are the dependencies all on public packages, or internal ones? (The leading question here is eventually: can you provide

pex3 lock create

command that shows the slowness?)

Sometimes 20 minutes + and that is also with the additional step Julie shared above in which I’m compiling a constraints.txt so there is a set range of hashed packages being assessed by

pants generate-lockfiles

. It is one single resolve. We have ~ 320 dependencies so quite a few. ~ 5 of these are private packages that come from 6 different private pypi repos (which I have an inkling is part of the reason for slow locking i.e. having to query all of these endpoints. I’ve tried workarounds but always hit issues with multi-platform compatibility or differences in querying private repos.

If you can provide enough public details (above) I can try to help with (A), and I know you are aware of the in flight work for (C).

I will try out some of those profiling steps, thanks for the info.

Yes although the issue we had was on Pants v2.17 before mandatory lockfiles so “internal lockfiles for tools” may not be the quite correct term. I understand that on this version these lockfiles do not support Python 3.12?

I think

install_from_resolve

was around 2.17 and the old mechanism was removed around 2.18. The modern tool lockfiles in 2.32 support 3.9-3.14, 3.9 will probably be dropped Soon, and 3.15 will be added around the end of the year after the 3.15.0 release. But with

install_from_resolve

you can always do your own thing.

hat come from 6 different private pypi repos (which I have an inkling is part of the reason

This seems plausible if said internal packages are large or the repos don't support PEP 658. If the private package <-->private repo mapping is simple than resolves_to_sources <https://www.pantsbuild.org/dev/reference/subsystems/python#resolves_to_sources> might (this is speculation) be worth trying.

That could be worth a look, thanks.

I think

install_from_resolve

was around 2.17 and the old mechanism was removed around 2.18. The modern tool lockfiles in 2.32 support 3.9-3.14, 3.9 will probably be dropped Soon, and 3.15 will be added around the end of the year after the 3.15.0 release. But with

install_from_resolve

you can always do your own thing.

I am running into another problem now which is simply our fault. After bumping to

2.32

which happens with both a proper lockfile and with disabling resolves and targeting an existing

constraints.txt

which is that one of our private packages doesn’t have a complete RECORD file…

@hundreds-carpet-28072 what build system do you use for your private package that does not produce a proper RECORD? As far as I'm aware, all the standard ones out there do.

what build system do you use for your private package that does not produce a proper RECORD? As far as I’m aware, all the standard ones out there do.

A bespoke one that has outlasted its utility, unfortunately. I’ve made changes to include compliant RECORD files for the time being. That’s not anybody else’s problem other than us.