I have a python_awslambda question: I have a large number of functions with the same third-party dependencies, and prior to using pants was using Lambda Layers to avoid reuploading 100MB of deps for every function. In python_awslambda, the deps get built into the pex so I'm back to repeating lots of large uploads. Is there a way to filter a third-party dep out of the pex being sent to lambda?

I don’t have any first hand experience with this, but it looks like it is this you want to use: https://github.com/pantsbuild/pants/pull/13894

ah, yeah I think that's what I need, let me see if it generates the plain pex without the deps I'm trying to avoid

To point out another possibility, you might be able to create a single lambdex with all your functions and then just alter the

LAMBDEX_ENTRY_POINT

in a tiny layer: https://github.com/pantsbuild/lambdex#controlling-runtime-execution

I see - so I would still upload my large deps in the pex, but include that env var to change it around

Yes. I re-read though and may have missed the point. You said Lambda Layers and not Docker layers so my mental model is probably wrong. I have no clue what Lambda Layers are.

they're fairly close, but you can have lambda layers added to a function after you've deployed it, and reuse the layer using its ARN across many lambdas. In our case we have some fairly hefty C dependencies that change infrequently. To make deploys fast & not reupload the C deps every time we deploy, we put them in a layer

Gotcha.

I made a PR of the change to add

include_requirements

as a field on python_awslambda https://github.com/pantsbuild/pants/pull/14469

Automatically. Targets and Subsystems get generated docs - everything in the Reference section of https://www.pantsbuild.org/. All the rest of the docs you use the web UI to suggest edits.

ah that's so neat

Thanks for digging in here!

of course 🙂 I'm not entirely sure how to test this PR without just stubbing everything, since it's just a pass of bool flags through to PexTarget

If you feel inspired, the infrastructure is all there https://github.com/pantsbuild/pants/blob/4b335cdccfebcbd1f92caaa5fef7d5ca161c7226/src/python/pants/backend/awslambda/python/rules_test.py#L76. I think it would be pretty easy to add a single pinned dep (say ansicolors) and check include / exclude.

When I try to run that test I get this error (macos, I think it may be that I'm missing python3.7 development headers?):

E   ImportError: dlopen(/private/var/folders/_s/q_xksw8943540mqd5vllmmgh0000gn/T/process-executionO6ryWQ/src/python/pants/engine/internals/native_engine.so, 0x0002): symbol not found in flat namespace '_PyObject_VectorcallMethod'
ImportError while importing test module '/private/var/folders/_s/q_xksw8943540mqd5vllmmgh0000gn/T/process-executionO6ryWQ/src/python/pants/backend/awslambda/python/rules_test.py'.
Hint: make sure your test modules/packages have valid Python names.

You can force the test to use a different Python version by using

--python-interpreter-constraints=['==3.9.*']

here https://github.com/pantsbuild/pants/blob/4b335cdccfebcbd1f92caaa5fef7d5ca161c7226/src/python/pants/backend/awslambda/python/rules_test.py#L55

Wouldn't that mean I have to transitive exclude every dependency though, and remember all future ones?

indeed, so your PR is a much better solution in the case that you want to leave off every 3rd-party requirement. Whereas

!!

transitive excludes are good if you only want to exclude a few expensive deps, but keep others

Gotcha - I'll add that alternative to the PR description

This is great, thanks for the PR! Just to clarify though, this is an all-or-nothing proposition: it will omit all 3rd party requirements from the lambda, so you have to provide them all in a layer. As long as this solves the problem, great! A more selective solution would require more work (such as implementing https://docs.google.com/document/d/1r9nCf62lHeaz5NGldhsZwu19_SzRuTCrPEELgn_9l2g/edit?usp=sharing)

Correct it does omit all 3rd party requirements

Hah, yeah it's a deep cut

yeah that's all outside pants right now, I'm just starting to adopt it and I already have a generation step for layers in my CI

BTW you can somewhat simplify that invocation with

/pants dependencies share/deployer/:: | xargs ./pants peek | jq -r '.[].requirements|select(.!=null)[]'

oh. I ripped it from the suggested

build-support

file

Yeah I only realized this myself a few days ago

neat

It is probably more future-proof!