Is there a way to take a pex_binary target and cre...
# general
b
Is there a way to take a pex_binary target and create a requirements.txt file from it? It's a follow up from a previous question - but I'm trying to cross-platform build docker images and it is really really slow. Like 100x+ slower when building arm from x86. I'm wondering if it would be easier to download all packages at runtime instead. I want the packages from a pex compile to do that though. Will take any advice.
c
To literally generate a requirements.txt you could -- among various PEX_TOOLS tricks -- run
pip freeze
https://github.com/pantsbuild/pants/discussions/20368 For what you are trying to do though, there are several variant of using
complete_platform
to assemble an x86_64 pex on arm64, or vise versa. You could then, for example, use copy the right wheels around. (The details are obviously specific to what you are trying to do, but in general if one is trying to deploy on arch X, it would be advisable to also test/build on arch X too.)
b
complete_platform gets me the whls, but running pex compile in a
docker_image
rule for a cross platform build is obnoxiously slow. Usually we're running on x86, but we also need to support arm builds soon. I rebuilt all our deps for arm on an arm machine and even made an arm deployment work, but most people are working on x86 machines and need the ability to test runs on x86 or arm.
There are a lot of options here like • packaging pants into the docker image and building the pex env remotely • staging the pex env and pulling it in • etc However, one of the other issues coming up is we are testing out ray and I can't easily use an entrypoint to pull in the pex for the ray workers pool on startup if I try to delay any installation to the entrypoint. It's kinda messy. There's also one other complication to the situation I can't mention.
This is unfortunately a mess šŸ˜ž
g
https://www.pantsbuild.org/blog/2022/08/02/optimizing-python-docker-deploys-using-pants but build your deps and publish, then the second layer always pull the deps layer and stack local code on top.
b
We already do all of that šŸ™‚ Took about 1.5 hours to build the largest intermediate image. Trying to figure it out if that's okay for us.
b
To answer the OP directly - lots of ways:
Copy code
# Using external pex-tools:
:; pex requests -o requests.pex

:; pex-tools requests.pex repository info -v | jq -r '.project_name + "==" + .version'
requests==2.32.5
charset-normalizer==3.4.4
idna==3.11
urllib3==2.5.0
certifi==2025.10.5

# Using embedded pex tools (comes along with --venv PEXes for free as well)
:; pex requests -o requests.pex --include-tools

:; PEX_TOOLS=1 ./requests.pex repository info -v | jq -r '.project_name + "==" + .version'
requests==2.32.5
charset-normalizer==3.4.4
idna==3.11
urllib3==2.5.0
certifi==2025.10.5

# No pex tools at all
:; unzip -qc requests.pex PEX-INFO | jq -r '.distributions | keys[]' | cut -d- -f1-2 | sed -e 's|-|==|'
certifi==2025.10.5
charset_normalizer==3.4.4
idna==3.11
requests==2.32.5
urllib3==2.5.0
complete_platform gets me the whls
@brief-engine-92399 using
--complete-platforms
means you shouldn't need to run Pex in a docker container at all.
b
Per the blog above, we're still running pex compile in the container build steps. I think that's causing some of the slowness I'm seeing
b
For example, see here: https://github.com/pex-tool/pex/issues/2949#issuecomment-3408725425 Build an aarch64 Linux PEX on x86_64 Linux.
b
This is for a comically large image
b
I saw that, but if your image is running under emulation, that alone explains slowness of running anything at all.
qemu is not fast!
b
I know. Was asking here hoping for a better solution šŸ˜ž
I kinda anticipated this would be the response.
b
I am giving that! Do not use a container
Just use --complete-platforms on the native host
You can't always get away with
--complete-platforms
, but hen you can, its good.
b
So we can build the pex env, but we need the container for orchestration purposes. We can't do anything like stage the pex and then pull it in during startup time for a silly reason I can't get into.
b
Alrighty then. Well you will have no solutions if you're forced to build the PEX in the container.
šŸ˜ž 1
I mean, the same solutions as outside. If the slowest thing is zipping, and zipping is 100x slower in the container - don;t zip, use --no-compress, etc.
or --no-pre-install-wheels
But those are just the same Pex options you can use on a native host, just savings will be magnified for CPU intensive operations since those are extra slow under qemu.