https://pantsbuild.org/ logo
#general
Title
# general
r

ripe-cpu-85141

08/09/2022, 11:01 PM
Hello, I have an issue related to lambdex. I create a lambda via the target python_awslambda with the runtime python3.9. I need a recent boto version and I can see in the zip file that the latest version is there (1.24.47). But if I run the lambda and ask for the version with
boto3.__version__
I get 1.20.32, which I think is the boto3 version included in the runtime. Is there a way to force the usage of the packaged boto3 over the one on the runtime?
h

hundreds-father-404

08/09/2022, 11:04 PM
That sounds right -- if you look at https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html, it shows that the runtime provides 1.20.32
I don't have a ton of AWS Lambda experience, but suspect you may need to do something like https://docs.aws.amazon.com/lambda/latest/dg/runtimes-modify.html or https://docs.aws.amazon.com/lambda/latest/dg/runtimes-custom.html?
r

ripe-cpu-85141

08/09/2022, 11:15 PM
well, this looks a bit overkill. I would consider that packages packaged with the lambdex zip files should have precedence over the runtime thing
h

hundreds-father-404

08/09/2022, 11:16 PM
I think that's entirely up to AWS Lambda. Pants can't control how the zip file gets consumed by them. *But I could be wrong, I don't have much experience here
r

ripe-cpu-85141

08/09/2022, 11:16 PM
If I do a dirty zip file with the boto3 package install at the root dir, it will be loaded instead of the one from the runtime
for what I see, nothing loads boto3 before I do. So maybe it's just a matter of setting a PYTHONPATH?
I'll try to investigate but I don't know the lambdex/pex code very well 🙂
h

hundreds-father-404

08/09/2022, 11:29 PM
That sounds plausible, although this is definitely out of my forte. It sounds likely this needs to be changed on the AWS side rather than Pex/Lambdex/Pants side, i.e. in step 4 here: https://www.pantsbuild.org/docs/awslambda-python#step-4-upload-to-aws
r

ripe-cpu-85141

08/10/2022, 1:36 AM
pex seems to append to the sys.path. I think it should prepend instead.
h

hundreds-father-404

08/10/2022, 1:41 AM
This exists for `pex_binary`: https://www.pantsbuild.org/v2.13/docs/reference-pex_binary#codeinherit_pathcode I don't see it for
python_awslambda
, but it should be easy to add. Interested in opening a PR?
r

ripe-cpu-85141

08/10/2022, 1:47 AM
so the system with the .deps directory and stuff are identical for lambda and pex?
*pex binaries
h

hundreds-father-404

08/10/2022, 1:49 AM
yeah, Lambdex spits out a Pex file that ends in the extension
.zip
rather than
.pex
if you look at the plugin code for
package_pex_binary.py
and
python/awslambda/rules.py
, they are extremely similar
h

happy-kitchen-89482

08/10/2022, 1:51 AM
Pants uses a tool called
lambdex
that takes a regular .pex and emits a .zip file that is essentially the same pex, but whose entry point is a valid lambda handler function, and some machinery to glue that to the original pex entry point.
1
So it's pex all the way down
r

ripe-cpu-85141

08/10/2022, 1:54 AM
ah ok, I see the same structure in a pex file. Sorry I'm still discovering how all that works
h

hundreds-father-404

08/10/2022, 1:58 AM
Nothing to apologize for! This isn't very obvious stuff, and we're happy to explain 🙂
r

ripe-cpu-85141

08/10/2022, 2:02 AM
I'm ok with making a PR but meanwhile, I need to do something to fix my project; this is the only thing remaining for me to switch and go to prod. Do you think I can make a plugin with an "enhanced" target based on the step you provided?
h

hundreds-father-404

08/10/2022, 2:03 AM
Yes, for sure that should work. The quickest and dirtiest thing to do is disable the builtin AWS backend, copy and paste all of pantsbuild/pants's version, and apply the changes I suggested. Follow the guide at https://www.pantsbuild.org/docs/plugins-overview for how you make an in-repo plugin
when copying-and-pasting, you'll want to copy the same version as your
pants_version
in
pants.toml
h

happy-kitchen-89482

08/10/2022, 2:08 AM
It’s been on my mental TODO list to investigate dropping lambdex in favor of creating lambdas with the “conventional” structure. Or rather, to investigate if doing so would have performance benefits re keeping executors warm. We’ve had reports that lambdex-created lambdas incur greater startup overhead in the warm scenario. So this issue could be another reason to investigate that. Then adding prepend ability wouldn’t be necessary, as we’d be creating completely normal lambdas.
👀 1
h

hundreds-father-404

08/10/2022, 2:09 AM
maybe create a ticket to track that?
r

ripe-cpu-85141

08/10/2022, 2:48 AM
aah, it works 🎉
❤️ 1
e

enough-analyst-54434

08/10/2022, 5:27 PM
We’ve had reports that lambdex-created lambdas incur greater startup overhead in the warm scenario.
It would be great if there were an issue for this, but this is "expected". A standard lambda has only /tmp write access and that storage is ephemeral; so PEX_ROOT gets redirected there and then promptly nuked after the lambda finishes. Each lambda run incurs what is normally a 1-time PEX extract cost. To make a PEX fast on lambda requires mounting EFS and configuring a PEX_ROOT env var to point to it or else using a container fwict. ... so re-working the Pants lambda support seems like a good idea.
And, sorry to be late to the party here, but no custom plugin is needed - just define the `PEX_INHERIT_PATH`environment variable in your lambda config on AWS and set it to
fallback
. I was able to repro, then correct using this just now.
💯 1
🙏 1
image.png,image.png
h

hundreds-father-404

08/10/2022, 5:45 PM
ohhh good point. Thoughts on adding it still as a contribution to Pants?
e

enough-analyst-54434

08/10/2022, 5:49 PM
It seems to make little sense to me. Using the env var works and eaither way you need docs which is the key bit. If the docs say set this pants thing they could instead just say set this AWS thing. And the latter is more flexible - there are other PEX_ env vars you might want to set out on AWS. What does make sense as a contribution is what Benjy spoke about. PEX gets foiled by no ephemeral storage only which is the default out on a lambda, and adding non-ephemeral cost the end user $$$.
h

happy-kitchen-89482

08/10/2022, 5:53 PM
Yeah, probably lambdex is only a good fit for one-off lambdas
not for ones where warm startup time matters
r

ripe-cpu-85141

08/12/2022, 3:43 PM
Ah thanks for the
PEX_INHERIT_PATH
, I didn't think to check that. To be honest, I considered the pants target to be my interface, I didn't think of looking at the pex level. But it makes sense. 🙂
e

enough-analyst-54434

08/12/2022, 3:54 PM
The relevant tools to learn about your PEX are
pex --help-variables
and
PEX_TOOLS=1 ./your.pex
. The latter only works if you use `pex_binary(include_tools=True)`or `pex_binary(execution_mode="venv")`but you can also
pex-tools ./your.pex
.
4 Views