Trying to create a dry-run equivalent 🤷

I'm using pex3 directly to create the lockfiles, but I'm messing with interpreter constraints here (tried to pull this directly from a

pants

call, but it may not be a perfect 1:1 of what's happening in

pants

):

pex3 lock create \
    --no-emit-warnings \
    --output baseline-3.12.json \
    --pip-version "24.1.2" \
    --resolver-version pip-2020-resolver \
    --style universal \
    --target-system linux \
    --target-system mac \
    --indent 2 \
    --manylinux manylinux2014 \
    --interpreter-constraint "CPython==3.12.*" \
    fastapi numpy pandas pydantic scipy sqlalchemy uvicorn

So.... This is one of those things that seems obvious when I run the test, but it's definitely never top of mind while trying to help people with these problems - because Pants kinda magicks it all away. I'm not sure if the results make sense from a

pex

point of view, but intuitively, I would assume that broader interpreter-constraints would take more time to resolve 🤷

Cold run 3.12.*:   0m18.203s
No-op run 3.12.*:  0m6.207s

Cold run 3.12.4:   0m15.587s
No-op run 3.12.4:  0m3.531s

Cold run >=3.9,<3.13:  0m48.746s
No-op run >=3.9,<3.13: 0m37.877s

If my results are reproducible by others, I think this would explain why my lockfile generation is pretty quick. I lock down my constraints pretty tightly in production code, and I usually know exactly the interpreter I plan to run against (unlike in my test/sample code, where I do whatever)

Lockfile generation in the st2 repo takes 5-10 min iirc.

Like the torch repo 🙂

I've ripped out enough references to our internal cheese shop and various forks that this should be runnable. It takes around 15-30 minutes.

Yeah, this bombs out for me on pyspark. I am getting a nice appreciation of what people might be going through while debugging - this is infuriating

I think this helps a bit with early failures

[python]
...
pip_version = "24.2"

[pex-cli]
version = "v2.13.0"
known_versions = [
  "v2.13.0|macos_arm64|dba08dde901cfbc43176a7d99414c7c58a3eeaa41743d2abbdf41a58d1d5ead9|4190750",
  "v2.13.0|macos_x86_64|dba08dde901cfbc43176a7d99414c7c58a3eeaa41743d2abbdf41a58d1d5ead9|4190750",
  "v2.13.0|linux_x86_64|dba08dde901cfbc43176a7d99414c7c58a3eeaa41743d2abbdf41a58d1d5ead9|4190750",
  "v2.13.0|linux_arm64|dba08dde901cfbc43176a7d99414c7c58a3eeaa41743d2abbdf41a58d1d5ead9|4190750"
]

uv

crushed that

uv pip install

- took like 20 seconds

I have to concede, in this quick test, I can't claim to know what pip was doing about downloading - as in, maybe it was downloading in parallel - but just emitting to the CLI in a sequential fashion, or maybe it was legitimately going one by one

uv has the trick where you only download partial pieces of the packages

Range requests?

I couldn't quite put it together in the 10 minutes I gave it, but I was thinking of something like this

Yeah, this is an interesting idea - I just did it, and the subsequent pip install was pretty quick (comparatively, obviously)

🤣 So close to being awesome...

uv pip compile requirements.in | grep -v "#" > requirements.txt
parallel -a requirements.txt -j12 pip install

I'm currently playing around with hacks, not solutions, e.g. could we use uv to pre-download deps and then have pex use them as cached, and other shenanigans

@powerful-scooter-95162 How about this?

file(
    name="requirements",
    source="<http://requirements.in|requirements.in>", 
)

system_binary(
    name="uv",
    binary_name="uv",
    fingerprint_args=["--version"],
)

run_shell_command(
    name="lock",
    command="uv pip compile <http://requirements.in|requirements.in> -o didthiswork.txt",
    runnable_dependencies=[":uv"],
    execution_dependencies=[":requirements"],
)

Then, maybe an alias that runs

pants run :lock

followed by

generate-lockfiles

?

You might not even need the

execution_dependencies

would have been neat to wire it as a dep of the python_requirements target, but I guess that's actually not necessary at all