Hello, I have an issue related to lambdex. I create a lambda via the target python_awslambda with the runtime python3.9. I need a recent boto version and I can see in the zip file that the latest version is there (1.24.47). But if I run the lambda and ask for the version with

boto3.__version__

I get 1.20.32, which I think is the boto3 version included in the runtime. Is there a way to force the usage of the packaged boto3 over the one on the runtime?

That sounds right -- if you look at https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html, it shows that the runtime provides 1.20.32

well, this looks a bit overkill. I would consider that packages packaged with the lambdex zip files should have precedence over the runtime thing

I think that's entirely up to AWS Lambda. Pants can't control how the zip file gets consumed by them. *But I could be wrong, I don't have much experience here

If I do a dirty zip file with the boto3 package install at the root dir, it will be loaded instead of the one from the runtime

That sounds plausible, although this is definitely out of my forte. It sounds likely this needs to be changed on the AWS side rather than Pex/Lambdex/Pants side, i.e. in step 4 here: https://www.pantsbuild.org/docs/awslambda-python#step-4-upload-to-aws

pex seems to append to the sys.path. I think it should prepend instead.

This exists for `pex_binary`: https://www.pantsbuild.org/v2.13/docs/reference-pex_binary#codeinherit_pathcode I don't see it for

python_awslambda

, but it should be easy to add. Interested in opening a PR?

so the system with the .deps directory and stuff are identical for lambda and pex?

yeah, Lambdex spits out a Pex file that ends in the extension

.zip

rather than

.pex

Pants uses a tool called

lambdex

that takes a regular .pex and emits a .zip file that is essentially the same pex, but whose entry point is a valid lambda handler function, and some machinery to glue that to the original pex entry point.

ah ok, I see the same structure in a pex file. Sorry I'm still discovering how all that works

Nothing to apologize for! This isn't very obvious stuff, and we're happy to explain 🙂

I'm ok with making a PR but meanwhile, I need to do something to fix my project; this is the only thing remaining for me to switch and go to prod. Do you think I can make a plugin with an "enhanced" target based on the step you provided?

Yes, for sure that should work. The quickest and dirtiest thing to do is disable the builtin AWS backend, copy and paste all of pantsbuild/pants's version, and apply the changes I suggested. Follow the guide at https://www.pantsbuild.org/docs/plugins-overview for how you make an in-repo plugin

It’s been on my mental TODO list to investigate dropping lambdex in favor of creating lambdas with the “conventional” structure. Or rather, to investigate if doing so would have performance benefits re keeping executors warm. We’ve had reports that lambdex-created lambdas incur greater startup overhead in the warm scenario. So this issue could be another reason to investigate that. Then adding prepend ability wouldn’t be necessary, as we’d be creating completely normal lambdas.

maybe create a ticket to track that?

aah, it works 🎉

We’ve had reports that lambdex-created lambdas incur greater startup overhead in the warm scenario.

It would be great if there were an issue for this, but this is "expected". A standard lambda has only /tmp write access and that storage is ephemeral; so PEX_ROOT gets redirected there and then promptly nuked after the lambda finishes. Each lambda run incurs what is normally a 1-time PEX extract cost. To make a PEX fast on lambda requires mounting EFS and configuring a PEX_ROOT env var to point to it or else using a container fwict. ... so re-working the Pants lambda support seems like a good idea.

ohhh good point. Thoughts on adding it still as a contribution to Pants?

It seems to make little sense to me. Using the env var works and eaither way you need docs which is the key bit. If the docs say set this pants thing they could instead just say set this AWS thing. And the latter is more flexible - there are other PEX_ env vars you might want to set out on AWS. What does make sense as a contribution is what Benjy spoke about. PEX gets foiled by no ephemeral storage only which is the default out on a lambda, and adding non-ephemeral cost the end user $$$.

Yeah, probably lambdex is only a good fit for one-off lambdas

Ah thanks for the

PEX_INHERIT_PATH

, I didn't think to check that. To be honest, I considered the pants target to be my interface, I didn't think of looking at the pex level. But it makes sense. 🙂

The relevant tools to learn about your PEX are

pex --help-variables

and

PEX_TOOLS=1 ./your.pex

. The latter only works if you use `pex_binary(include_tools=True)`or `pex_binary(execution_mode="venv")`but you can also

pex-tools ./your.pex

.