Is there a way to skip the building of a custom pex when running python targets locally? While building a small pex makes sense for deployment, to the extent that everything comes from a single resolve, having a single venv/pex for that resolve seems like it would speed up workflows that run a bunch of targets

Can you use

pants export

to create a venv from your resolve that includes all dependencies? That might be a way to get around this issue

probably, is that the common pattern for local development?

Yeah, that’s what I do - export a venv with all dependencies for local dev, and then run scripts using Python directly (not Pants) inside that venv

sort of odd that the default way of running things from a build tool isn't optimized for local execution though

If you want a single pex used for all targets you may run for a resolve, you may want to set this to true: https://www.pantsbuild.org/2.20/reference/subsystems/python#run_against_entire_lockfile I don’t consider the export option to be the default way of running things, but it’s a nice escape hatch when the default doesn’t do what you want..

If you run a python source file (

pants run path/to/main.py

) instead of a pex_binary target) then the pex packaging is skipped. See here for details

Thanks for the pointers, the entire lock file worked (at the obvious cost of taking longer to build it, which is expected), but I do see pexes being built for running python targets:

$ pants --no-pantsd --no-local-cache run billing/update_billing_data.py -- --user=...
10:56:50.03 [INFO] Canceled: Building setuptools_scm.pex from <resource://pants.backend.python.subsystems/setuptools_scm.lock>
10:56:51.70 [INFO] Completed: Building local_dists.pex
10:56:51.86 [INFO] Completed: Building setuptools_scm.pex from <resource://pants.backend.python.subsystems/setuptools_scm.lock>
10:56:52.08 [INFO] Completed: Run setuptools_scm for libs:_version
⠈ 2.49s Building 13 requirements for billing.pex from the 3rdparty/python/default.lock resolve: httpx==0.27.0, numpy==1.26.4, opentelemetry-sdk
10:56:54.93 [INFO] Canceled: Building 13 requirements for billing.pex from the 3rdparty/python/default.lock resolve: httpx==0.27.0, numpy==1.26.4, opentelemetry-sdk==1.24.0; python_version >= "3.8", pandarallel==1.6.5, pandas-stu... (227 characters truncated)

Setting

python_sources(run_goal_use_sandbox=False)

Does not change the billing.pex being constructed

That's the "requirements pex" - containing just the 3rdparty requirements. That does have to be built, but it is only rebuilt when the 3rdparty requirements change, so it's usually cached. The big perf win is that we don't build the full final pex that includes your sources, because those change often and that pex would keep having to be rebuilt otherwise

@happy-kitchen-89482 -- in my experience, the caching of the built requirements pex is broken. This PR is a stab in the dark attempt at potentially address this if it's from an unsorted list of things going into the cache key.

yeah, my experience is also that it doesn't get cached fairly often with no 3rd party changes

Ooof, well that is clearly a bug