Hello - I'm trying to create an AWS Lambda zip fil...
# general
Hello - I'm trying to create an AWS Lambda zip file. Im running my command to create the zip on an EC2 ubuntu instance since my local machine is a Mac. However, I'm getting this error when I run
~/bin/pants package project/lambda_example.py
. Any ideas?
Copy code
21:23:38.17 [ERROR] 1 Exception encountered:

Engine traceback:
  in `package` goal
  in Create Python AWS Lambda - project:lambda

ProcessExecutionFailure: Process 'Building lambdex.pex from lambdex_default.lock' failed with exit code 1.

There were 2 errors downloading required artifacts:
1. lambdex 0.1.9 from <https://files.pythonhosted.org/packages/20/89/05c73da172c510299869e904543825222ffa990b719ffaba4bce2d1be0c4/lambdex-0.1.9-py2.py3-none-any.whl>
    ModuleNotFoundError: No module named 'distutils.util'
2. pex 2.1.111 from <https://files.pythonhosted.org/packages/5d/08/89438cc626ec77a6f7dc3ab17cad581d14882a2f51a37d24c8a4c127e7bc/pex-2.1.111-py2.py3-none-any.whl>
    ModuleNotFoundError: No module named 'distutils.util'
You likely need to install
with apt.
I've run
sudo apt-get install python3-distutils
Some more context: the ubuntu instance came with python3.10 installed, but it didn't seem like the lambda plugin supported that, so i installed python3.9 and bumped my version in BUILD down to 3.9
Great context! What about
apt install python3.9-distutils
? 🙂
let me try that
that worked! for python3.8. I have python3.9 installed as well but that didn't work so I bumped it down to 3.8
That likely means your pants itself is running with 3.8... Are you running the new "get-pants.sh" pants or the old
? I assume the latter, but just to double-check.
I had curled the install script > get-pants.sh, then installed it. this was before i installed python 3.8 or python 3.9 (back when only 3.10 was installed by default)
Interesting. I'm a bit out of my depth re: that then, I thought that the new one would always run 3.9. Maybe the pex builder uses the host Python still.
Also - I updated my lambda function using the lambda.zip and tried to trigger an invocation using the "Test" button. I get the error
Copy code
"Unable to import module 'lambda_function': No module named 'lambda_function'"
Am I supposed to name my lambda handler file and handler function a certain name? I'm just following the instructions here: https://www.pantsbuild.org/docs/awslambda-python and the file is named "lambda_example.py" and the function is named "example_handler.py" ; seems like my Lambda function couldn't find those or know about those?
By the way I would have used Ubuntu standard:5.0 since Ubuntu standard:6.0 only supports 3.10. I am using standard:5.0 and since it has pyenv, I manually installed python 3.10 inside it. Can you write here how you are defining your
target? Your handler should look like
as described here https://www.pantsbuild.org/docs/reference-python_awslambda#codehandlercode
I copied and pasted the BUILD and lambda_example.py here: https://www.pantsbuild.org/docs/awslambda-python
oh im a dumbass - i need to edit my lambda function's runtime settings to use the handler python file name and python function name
😄 1
is the lambda integration for pants supposed to download the dependencies in my handler function as well and zip that up? I imported
but when I'm trying to run my function, it's saying that the module can't be foun
It will as long as you define it using one of the
function like
Copy code
and you should have
with your python packages in same directory where this is defined
is it possible to not define a requirements.txt or have pants auto-infer and generate that for me?
since the awslambda-python integration already is doing dependency inference based on my imports
as part of the package goal
"Pants will use dependency inference based on the
field, which you can confirm by running
pants dependencies path/to:lambda
No, I don’t think so. Pants only does auto-infer for the local imports. It still needs to know what 3rd party dependencies it needs from outside. Although you can give it 100 3rd party packages but it will only pick those which you have imported
^ how would you parse that above statement?
re: dependency inference from the pants docs
Read this https://www.pantsbuild.org/docs/targets#dependencies-field
A target’s dependencies determines which other first-party code and third-party requirements to include when building the target.
Usually, you leave off the
field thanks to dependency inference. Pants will read your import statements and map those imports back to your first-party code and your third-party requirements. You can run
pants dependencies path/to:target
to see what dependencies Pants infers.
hm. what is the point of the auto-infer for local imports then? im still not quite sure what local imports is referring to
So if you have modules or packages locally called
Copy code
import local_module #1st party
import pandas as pd #3rd party
i see. so a way to get around this would be to call
pants dependencies
to get the list of 3rd party modules that are inferred, add that to the dependencies in my BUILD, then call
pants package
No. It’s different than that. You should definitely read the docs. I feel like I might put something wrong in your head if I go too deep explaining it.
👍 1
I went through the docs, and things i ended up tweaking were: • added a requirements.txt and an entry in BUILD:
Copy code
python_requirements( source="requirements.txt"
• In my Lambda runtime settings, I had to set the handler to the auto-generated file and function name:
Still a bit confused about the 3rd party inference part for inferring pandas and numpy, as the docs makes it sound like those are auto-inferred
I think there's two different concepts of dependency here that are confusing: • In terms of a target/build it depends on other things in Pants • What do you depend on from outside Pants The first kind is inferred, the second one is not.
bridges these, making Pants know about more things that it can infer from. So: • add
to `requirements.txt`: Pants learns that pandas exists, where it comes from, etc • add
to a directory: Pants learn those files exist • when looking at a new file it'll see what it imports and try to find those from the things it knows about from before
💯 1