Hello, I have an issue related to lambdex. I creat...
# general
r
Hello, I have an issue related to lambdex. I create a lambda via the target python_awslambda with the runtime python3.9. I need a recent boto version and I can see in the zip file that the latest version is there (1.24.47). But if I run the lambda and ask for the version with
boto3.__version__
I get 1.20.32, which I think is the boto3 version included in the runtime. Is there a way to force the usage of the packaged boto3 over the one on the runtime?
h
That sounds right -- if you look at https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html, it shows that the runtime provides 1.20.32
I don't have a ton of AWS Lambda experience, but suspect you may need to do something like https://docs.aws.amazon.com/lambda/latest/dg/runtimes-modify.html or https://docs.aws.amazon.com/lambda/latest/dg/runtimes-custom.html?
r
well, this looks a bit overkill. I would consider that packages packaged with the lambdex zip files should have precedence over the runtime thing
h
I think that's entirely up to AWS Lambda. Pants can't control how the zip file gets consumed by them. *But I could be wrong, I don't have much experience here
r
If I do a dirty zip file with the boto3 package install at the root dir, it will be loaded instead of the one from the runtime
for what I see, nothing loads boto3 before I do. So maybe it's just a matter of setting a PYTHONPATH?
I'll try to investigate but I don't know the lambdex/pex code very well 🙂
h
That sounds plausible, although this is definitely out of my forte. It sounds likely this needs to be changed on the AWS side rather than Pex/Lambdex/Pants side, i.e. in step 4 here: https://www.pantsbuild.org/docs/awslambda-python#step-4-upload-to-aws
r
pex seems to append to the sys.path. I think it should prepend instead.
h
This exists for `pex_binary`: https://www.pantsbuild.org/v2.13/docs/reference-pex_binary#codeinherit_pathcode I don't see it for
python_awslambda
, but it should be easy to add. Interested in opening a PR?
r
so the system with the .deps directory and stuff are identical for lambda and pex?
*pex binaries
h
yeah, Lambdex spits out a Pex file that ends in the extension
.zip
rather than
.pex
if you look at the plugin code for
package_pex_binary.py
and
python/awslambda/rules.py
, they are extremely similar
h
Pants uses a tool called
lambdex
that takes a regular .pex and emits a .zip file that is essentially the same pex, but whose entry point is a valid lambda handler function, and some machinery to glue that to the original pex entry point.
1
So it's pex all the way down
r
ah ok, I see the same structure in a pex file. Sorry I'm still discovering how all that works
h
Nothing to apologize for! This isn't very obvious stuff, and we're happy to explain 🙂
r
I'm ok with making a PR but meanwhile, I need to do something to fix my project; this is the only thing remaining for me to switch and go to prod. Do you think I can make a plugin with an "enhanced" target based on the step you provided?
h
Yes, for sure that should work. The quickest and dirtiest thing to do is disable the builtin AWS backend, copy and paste all of pantsbuild/pants's version, and apply the changes I suggested. Follow the guide at https://www.pantsbuild.org/docs/plugins-overview for how you make an in-repo plugin
when copying-and-pasting, you'll want to copy the same version as your
pants_version
in
pants.toml
h
It’s been on my mental TODO list to investigate dropping lambdex in favor of creating lambdas with the “conventional” structure. Or rather, to investigate if doing so would have performance benefits re keeping executors warm. We’ve had reports that lambdex-created lambdas incur greater startup overhead in the warm scenario. So this issue could be another reason to investigate that. Then adding prepend ability wouldn’t be necessary, as we’d be creating completely normal lambdas.
👀 1
h
maybe create a ticket to track that?
r
aah, it works 🎉
❤️ 1
e
We’ve had reports that lambdex-created lambdas incur greater startup overhead in the warm scenario.
It would be great if there were an issue for this, but this is "expected". A standard lambda has only /tmp write access and that storage is ephemeral; so PEX_ROOT gets redirected there and then promptly nuked after the lambda finishes. Each lambda run incurs what is normally a 1-time PEX extract cost. To make a PEX fast on lambda requires mounting EFS and configuring a PEX_ROOT env var to point to it or else using a container fwict. ... so re-working the Pants lambda support seems like a good idea.
And, sorry to be late to the party here, but no custom plugin is needed - just define the `PEX_INHERIT_PATH`environment variable in your lambda config on AWS and set it to
fallback
. I was able to repro, then correct using this just now.
💯 1
🙏 1
image.png,image.png
h
ohhh good point. Thoughts on adding it still as a contribution to Pants?
e
It seems to make little sense to me. Using the env var works and eaither way you need docs which is the key bit. If the docs say set this pants thing they could instead just say set this AWS thing. And the latter is more flexible - there are other PEX_ env vars you might want to set out on AWS. What does make sense as a contribution is what Benjy spoke about. PEX gets foiled by no ephemeral storage only which is the default out on a lambda, and adding non-ephemeral cost the end user $$$.
h
Yeah, probably lambdex is only a good fit for one-off lambdas
not for ones where warm startup time matters
r
Ah thanks for the
PEX_INHERIT_PATH
, I didn't think to check that. To be honest, I considered the pants target to be my interface, I didn't think of looking at the pex level. But it makes sense. 🙂
e
The relevant tools to learn about your PEX are
pex --help-variables
and
PEX_TOOLS=1 ./your.pex
. The latter only works if you use `pex_binary(include_tools=True)`or `pex_binary(execution_mode="venv")`but you can also
pex-tools ./your.pex
.