average-breakfast-91545
03/23/2023, 12:43 PMrequests
and structlog
to a layer, I need build a zip with the following structure:
python/
requests/
structlog/
Looks like the archive target only supports packages and files. Do I need to create a custom target for this? Is that something you'd like contributed back?enough-analyst-54434
03/23/2023, 12:48 PMpants package
a pex_binary(..., include_sources=False, include_tools=True)
2. Run PEX_TOOLS=1 my.pex venv create/it/here
3. Run mv create/it/here/lib/python3.10/site-packages python
4. Now zip up that python
dir.average-breakfast-91545
03/23/2023, 12:51 PMpython_lambda_layer(
dependencies=[
"3rdparty/python#requests",
"3rdparty/python#structlog"
])
and just package itenough-analyst-54434
03/23/2023, 12:53 PMaverage-breakfast-91545
03/23/2023, 12:53 PMbusy-vase-39202
03/23/2023, 1:25 PMaverage-breakfast-91545
03/25/2023, 10:56 AMdirutil
module that will create me a temp dir under the build root, but the docs are full of terrifying warnings about IO in rules. Is there something I need to be aware of here in order not to enrage the Cache Gods?enough-analyst-54434
03/25/2023, 11:23 AMaverage-breakfast-91545
03/25/2023, 11:28 AMenough-analyst-54434
03/25/2023, 11:33 AMaverage-breakfast-91545
03/25/2023, 11:34 AMenough-analyst-54434
03/25/2023, 11:35 AMaverage-breakfast-91545
03/25/2023, 11:35 AMenough-analyst-54434
03/25/2023, 11:36 AMaverage-breakfast-91545
03/25/2023, 11:36 AMenough-analyst-54434
03/25/2023, 11:37 AMaverage-breakfast-91545
03/25/2023, 11:38 AMOk, do you know what you're looking at in the . deps directory1 week of "experience" - I am firmly in the realm of unconscious incompetence
enough-analyst-54434
03/25/2023, 11:39 AMaverage-breakfast-91545
03/25/2023, 11:39 AMenough-analyst-54434
03/25/2023, 11:40 AMaverage-breakfast-91545
03/25/2023, 11:40 AMenough-analyst-54434
03/25/2023, 11:45 AMaverage-breakfast-91545
03/25/2023, 11:46 AMpython3.9
into the middle of a path, and handling arm runtimes, but it works, and I can run the resulting layer.
Outstanding questions:
IS there anything here that is grotesquely stupid? I've cobbled this together by reading rules in the existing pants codebase, mostly the export and package utils.
I'm pretty sure I don't need to take a PexPex
in my target, because I should just be able to run the requirements.pex, right? A quick test failed there because I didn't have PATH
set up in the env when running the process.
Is there a better way than this of getting a digest of installed requirements that I've requested in my target? The bit where we strip out the pex tools is a bit fugly and liable to break.
Idiomatically, would you break this down into multiple rules? It seems like I probably want a result type for LambdaVenvPex, and a result type for ZippedLambdaLayer, rather than a single massive script.enough-analyst-54434
03/26/2023, 3:32 PMPex
, but a VenvPex
instead since a Pex will yield a zipped up PEX which you immediately unzip - wasted work that can be significant for large ML deps. A VenvPex
will re-use the packed deps Pants has almost certainly already peddled in.
+ You should probably get the site-packages dir via something like a Process that runs ~ venv/bin/python -c 'import site; print(site.getsitepackages())'
+ You add --pip
only to turn around and strip out the tools adding it adds - you can probably just not add Pip.