https://pantsbuild.org/ logo
b

broad-processor-92400

03/31/2023, 7:22 AM
pip install
supports a
--target /path/to/dir
argument to install the reqs directly into that directory, without the usual venv structure. This is very handy for building lambda functions/layers, because doing that generally gives a directory that works as the root of a function/layer (e.g. it's what AWS's CDK does in their lambda packaging construct). Is there way to achieve something similar within when given a PEX file (or via the PEX CLI)? I'm going to experiment with
PEX_TOOLS=1 file.pex venv $(mktemp)
and using the contents of the
lib/python*/site-packages/
directory alone, dropping the rest of the venv structure, but that doesn't seem particularly right, on the surface...
a

average-breakfast-91545

03/31/2023, 8:36 AM
I've been doing some work on building lambda layers, doing more or less the same thing: Gather dependencies, install to a venv, copy the files back out to the
python
directory. https://github.com/bobthemighty/pants-lambda-layer/tree/master/pants-plugins I've not used it in anger yet, but it's worked in prototypes, and I'll probably be running it for real today.
šŸ‘ 1
e

enough-analyst-54434

03/31/2023, 8:44 AM
If you experiment with the PEX_TOOLS CLI you'll find
repository
subcommands. Now they don't help here, they only deal with 3rd party and the venv shenanigans deals with 1st and 3rd party. However, knowing those subcommands exist on top of
venv
pretty much makes the case that PEX tools should just support this case you both have directly.
šŸ‘ 1
b

broad-processor-92400

04/02/2023, 1:01 AM
I filed a feature request: https://github.com/pantsbuild/pex/issues/2110 It'd be handy to not have to run this within the appropriate AWS lambda builder image docker environment at build time; one way to avoid this might be via complete platform support for these export commands. Is that something that that is feasible/worth a feature request?
e

enough-analyst-54434

04/02/2023, 1:29 AM
That's included for free. The PEX you run the command against will either have been built with one or more complete platforms or it won't have, but that's orthogonal.
b

broad-processor-92400

04/02/2023, 2:27 AM
Ah, hm, I think I might be misunderstanding? I've built a pex for x86-64 linux including a native dep (e.g.
pydantic
), and cannot seem to use the tools to extract it when running them on macOS, or arm64 linux. https://gist.github.com/huonw/adbd1d685f595d10780e5f878a5245cd spells out what I tried. (To be clear, I'm willing to try the new environments to run the PEX within an appropriate image, so a feature here would be an optimisation.)
e

enough-analyst-54434

04/02/2023, 2:38 AM
I'm confused, why do the extraction on not the target image?
I think the extraction subcommand could take --platform / --complete-platform anyway if this is needed.
b

broad-processor-92400

04/02/2023, 2:45 AM
A few different reasons: ā€¢ slow along two dimensions (emulating an x86-64 image on an arm64 host or vice versa, unavoidable with mixed deploy targets, plus the Linux VM on a macOS host) ā€¢ environments are only recent/still experimental (e.g. I found a few reliability bugs with a few minutes of testing, and that makes me very nervous about forcing this upon other devs in our team as a critical part of our build pipeline) ā€¢ general fiddliness of ensuring docker is configured correctly for local dev (at least one person on our team is in a weird limbo state where docker works usually, but not always šŸ¤·ā€ā™‚ļø ) That said, we don't require movement on this in the near future, so happy to just wait and hope that environments bake a bit more before we do need it.
e

enough-analyst-54434

04/02/2023, 2:49 AM
This is definitely doable.
b

broad-processor-92400

04/02/2023, 2:51 AM
Is it reasonable, in addition to being doable?
e

enough-analyst-54434

04/02/2023, 4:47 AM
Yeah. The --python, --interpreter-constraint, --platform and --complete-platform arguments map to
Targets
internally, which is a standard input to both a Pip resolve and a boot resolve from a PEX. The default target is the current interpreter, you don't want that, you want the target(s) specified by those args, in particular --complete-platform.
šŸ‘ 1
b

broad-processor-92400

04/04/2023, 4:12 AM
2 Views