pip install

supports a

--target /path/to/dir

argument to install the reqs directly into that directory, without the usual venv structure. This is very handy for building lambda functions/layers, because doing that generally gives a directory that works as the root of a function/layer (e.g. it's what AWS's CDK does in their lambda packaging construct). Is there way to achieve something similar within when given a PEX file (or via the PEX CLI)? I'm going to experiment with

PEX_TOOLS=1 file.pex venv $(mktemp)

and using the contents of the

lib/python*/site-packages/

directory alone, dropping the rest of the venv structure, but that doesn't seem particularly right, on the surface...

I've been doing some work on building lambda layers, doing more or less the same thing: Gather dependencies, install to a venv, copy the files back out to the

python

directory. https://github.com/bobthemighty/pants-lambda-layer/tree/master/pants-plugins I've not used it in anger yet, but it's worked in prototypes, and I'll probably be running it for real today.

If you experiment with the PEX_TOOLS CLI you'll find

repository

subcommands. Now they don't help here, they only deal with 3rd party and the venv shenanigans deals with 1st and 3rd party. However, knowing those subcommands exist on top of

venv

pretty much makes the case that PEX tools should just support this case you both have directly.

I filed a feature request: https://github.com/pantsbuild/pex/issues/2110 It'd be handy to not have to run this within the appropriate AWS lambda builder image docker environment at build time; one way to avoid this might be via complete platform support for these export commands. Is that something that that is feasible/worth a feature request?

That's included for free. The PEX you run the command against will either have been built with one or more complete platforms or it won't have, but that's orthogonal.

Ah, hm, I think I might be misunderstanding? I've built a pex for x86-64 linux including a native dep (e.g.

pydantic

), and cannot seem to use the tools to extract it when running them on macOS, or arm64 linux. https://gist.github.com/huonw/adbd1d685f595d10780e5f878a5245cd spells out what I tried. (To be clear, I'm willing to try the new environments to run the PEX within an appropriate image, so a feature here would be an optimisation.)

I'm confused, why do the extraction on not the target image?

A few different reasons: • slow along two dimensions (emulating an x86-64 image on an arm64 host or vice versa, unavoidable with mixed deploy targets, plus the Linux VM on a macOS host) • environments are only recent/still experimental (e.g. I found a few reliability bugs with a few minutes of testing, and that makes me very nervous about forcing this upon other devs in our team as a critical part of our build pipeline) • general fiddliness of ensuring docker is configured correctly for local dev (at least one person on our team is in a weird limbo state where docker works usually, but not always 🤷‍♂️ ) That said, we don't require movement on this in the near future, so happy to just wait and hope that environments bake a bit more before we do need it.

This is definitely doable.

Is it reasonable, in addition to being doable?

Yeah. The --python, --interpreter-constraint, --platform and --complete-platform arguments map to

Targets

internally, which is a standard input to both a Pip resolve and a boot resolve from a PEX. The default target is the current interpreter, you don't want that, you want the target(s) specified by those args, in particular --complete-platform.

(I filed https://github.com/pantsbuild/pex/issues/2111)