https://pantsbuild.org/ logo
#general
Title
# general
q

quaint-oyster-63068

03/22/2023, 9:30 PM
Hello - I'm trying to create an AWS Lambda zip file. Im running my command to create the zip on an EC2 ubuntu instance since my local machine is a Mac. However, I'm getting this error when I run
~/bin/pants package project/lambda_example.py
. Any ideas?
Copy code
21:23:38.17 [ERROR] 1 Exception encountered:

Engine traceback:
  in `package` goal
  in Create Python AWS Lambda - project:lambda

ProcessExecutionFailure: Process 'Building lambdex.pex from lambdex_default.lock' failed with exit code 1.
stdout:

stderr:
There were 2 errors downloading required artifacts:
1. lambdex 0.1.9 from <https://files.pythonhosted.org/packages/20/89/05c73da172c510299869e904543825222ffa990b719ffaba4bce2d1be0c4/lambdex-0.1.9-py2.py3-none-any.whl>
    ModuleNotFoundError: No module named 'distutils.util'
2. pex 2.1.111 from <https://files.pythonhosted.org/packages/5d/08/89438cc626ec77a6f7dc3ab17cad581d14882a2f51a37d24c8a4c127e7bc/pex-2.1.111-py2.py3-none-any.whl>
    ModuleNotFoundError: No module named 'distutils.util'
g

gorgeous-winter-99296

03/22/2023, 9:33 PM
You likely need to install
python3-distutils
with apt.
q

quaint-oyster-63068

03/22/2023, 9:33 PM
I've run
sudo apt-get install python3-distutils
Some more context: the ubuntu instance came with python3.10 installed, but it didn't seem like the lambda plugin supported that, so i installed python3.9 and bumped my version in BUILD down to 3.9
g

gorgeous-winter-99296

03/22/2023, 9:34 PM
Great context! What about
apt install python3.9-distutils
? 🙂
q

quaint-oyster-63068

03/22/2023, 9:34 PM
let me try that
that worked! for python3.8. I have python3.9 installed as well but that didn't work so I bumped it down to 3.8
g

gorgeous-winter-99296

03/22/2023, 9:37 PM
That likely means your pants itself is running with 3.8... Are you running the new "get-pants.sh" pants or the old
./pants
? I assume the latter, but just to double-check.
q

quaint-oyster-63068

03/22/2023, 9:38 PM
I had curled the install script > get-pants.sh, then installed it. this was before i installed python 3.8 or python 3.9 (back when only 3.10 was installed by default)
g

gorgeous-winter-99296

03/22/2023, 9:41 PM
Interesting. I'm a bit out of my depth re: that then, I thought that the new one would always run 3.9. Maybe the pex builder uses the host Python still.
q

quaint-oyster-63068

03/22/2023, 9:41 PM
Also - I updated my lambda function using the lambda.zip and tried to trigger an invocation using the "Test" button. I get the error
Copy code
"Unable to import module 'lambda_function': No module named 'lambda_function'"
Am I supposed to name my lambda handler file and handler function a certain name? I'm just following the instructions here: https://www.pantsbuild.org/docs/awslambda-python and the file is named "lambda_example.py" and the function is named "example_handler.py" ; seems like my Lambda function couldn't find those or know about those?
r

refined-addition-53644

03/22/2023, 9:47 PM
By the way I would have used Ubuntu standard:5.0 since Ubuntu standard:6.0 only supports 3.10. I am using standard:5.0 and since it has pyenv, I manually installed python 3.10 inside it. Can you write here how you are defining your
python_awslambda
target? Your handler should look like
path.to.module:handler_func
as described here https://www.pantsbuild.org/docs/reference-python_awslambda#codehandlercode
q

quaint-oyster-63068

03/22/2023, 10:05 PM
I copied and pasted the BUILD and lambda_example.py here: https://www.pantsbuild.org/docs/awslambda-python
oh im a dumbass - i need to edit my lambda function's runtime settings to use the handler python file name and python function name
😄 1
is the lambda integration for pants supposed to download the dependencies in my handler function as well and zip that up? I imported
panads
but when I'm trying to run my function, it's saying that the module can't be foun
d
r

refined-addition-53644

03/22/2023, 10:24 PM
It will as long as you define it using one of the
_requirements
function like
Copy code
python_requirements()
and you should have
requirements.txt
with your python packages in same directory where this is defined
q

quaint-oyster-63068

03/22/2023, 10:24 PM
is it possible to not define a requirements.txt or have pants auto-infer and generate that for me?
since the awslambda-python integration already is doing dependency inference based on my imports
as part of the package goal
"Pants will use dependency inference based on the
handler
field, which you can confirm by running
pants dependencies path/to:lambda
"
r

refined-addition-53644

03/22/2023, 10:26 PM
No, I don’t think so. Pants only does auto-infer for the local imports. It still needs to know what 3rd party dependencies it needs from outside. Although you can give it 100 3rd party packages but it will only pick those which you have imported
q

quaint-oyster-63068

03/22/2023, 10:27 PM
^ how would you parse that above statement?
re: dependency inference from the pants docs
r

refined-addition-53644

03/22/2023, 10:29 PM
Read this https://www.pantsbuild.org/docs/targets#dependencies-field
A target’s dependencies determines which other first-party code and third-party requirements to include when building the target.
Usually, you leave off the
dependencies
field thanks to dependency inference. Pants will read your import statements and map those imports back to your first-party code and your third-party requirements. You can run
pants dependencies path/to:target
to see what dependencies Pants infers.
q

quaint-oyster-63068

03/22/2023, 10:34 PM
hm. what is the point of the auto-infer for local imports then? im still not quite sure what local imports is referring to
r

refined-addition-53644

03/22/2023, 10:35 PM
So if you have modules or packages locally called
local_module
Copy code
import local_module #1st party
import pandas as pd #3rd party
q

quaint-oyster-63068

03/22/2023, 10:37 PM
i see. so a way to get around this would be to call
pants dependencies
to get the list of 3rd party modules that are inferred, add that to the dependencies in my BUILD, then call
pants package
?
r

refined-addition-53644

03/22/2023, 10:39 PM
No. It’s different than that. You should definitely read the docs. I feel like I might put something wrong in your head if I go too deep explaining it.
👍 1
q

quaint-oyster-63068

03/22/2023, 11:08 PM
I went through the docs, and things i ended up tweaking were: • added a requirements.txt and an entry in BUILD:
Copy code
python_requirements( source="requirements.txt"
• In my Lambda runtime settings, I had to set the handler to the auto-generated file and function name:
lambdex_handler.handler
Still a bit confused about the 3rd party inference part for inferring pandas and numpy, as the docs makes it sound like those are auto-inferred
g

gorgeous-winter-99296

03/23/2023, 9:15 AM
I think there's two different concepts of dependency here that are confusing: • In terms of a target/build it depends on other things in Pants • What do you depend on from outside Pants The first kind is inferred, the second one is not.
python_requirements
bridges these, making Pants know about more things that it can infer from. So: • add
pandas
to `requirements.txt`: Pants learns that pandas exists, where it comes from, etc • add
python_sources()
to a directory: Pants learn those files exist • when looking at a new file it'll see what it imports and try to find those from the things it knows about from before
💯 1
2 Views