https://pantsbuild.org/ logo
Join Slack
Powered by
Hey Pants folks! I’m trying to reproduce the docum...
# general
s
Hey Pants folks! I’m trying to reproduce the documentation of using AWS Lambda and Docker with Pants, but I’m not able to get it running. When I try running my function I get:
Copy code
{
  "errorMessage": "the 'package' argument is required to perform a relative import for '.app.lambdex_handler'",
  "errorType": "TypeError",
  "requestId": "",
  "stackTrace": [
    "  File \"/var/lang/lib/python3.9/importlib/__init__.py\", line 122, in import_module\n    raise TypeError(msg.format(name))\n"
  ]
}
I have re-created the repo based on the docs here: https://github.com/nharada1/lambda-aws-pants-demo My only changes are I’m using Python 3.9 instead of 3.8
✅ 1
e
Pants is almost, but not quite, magic. Sometimes you need to teach it a few things about your project. As your project stands:
Copy code
$ ./pants package project:lambda
15:42:31.66 [INFO] Wrote dist/project/lambda.zip
    Runtime: python3.9
    Handler: lambdex_handler.handler
$ zipinfo dist/project/lambda.zip  | tail
-rw-r--r--  2.0 unx      156 b- defN 80-Jan-01 00:00 .bootstrap/pex/version.py
-rw-r--r--  2.0 unx      533 b- defN 80-Jan-01 00:00 PEX-INFO
-rwxr-xr-x  2.0 unx     3524 b- defN 80-Jan-01 00:00 __main__.py
drwxr-xr-x  2.0 unx        0 b- defN 80-Jan-01 00:00 __pex__/
-rw-r--r--  2.0 unx     3498 b- defN 80-Jan-01 00:00 __pex__/__init__.py
drwxr-xr-x  2.0 unx        0 b- defN 80-Jan-01 00:00 project/
-rw-r--r--  2.0 unx       60 b- defN 80-Jan-01 00:00 project/lambda_example.py
?rwxr-xr-x  2.0 unx       57 b- stor 80-Jan-01 00:00 LAMBDEX-INFO
?rwxr-xr-x  2.0 unx     1920 b- stor 80-Jan-01 00:00 lambdex_handler.py
134 files, 1749851 bytes uncompressed, 469448 bytes compressed:  73.2%
$ ./pants roots
.
See that 
project/lambda_example.py
? Not good. So teach Pants 
project/
is a project:
Copy code
$ git diff
diff --git a/pants.toml b/pants.toml
index 14052ad..eece8d2 100644
--- a/pants.toml
+++ b/pants.toml
@@ -10,4 +10,10 @@ backend_packages.add = [
 use_deprecated_directory_cli_args_semantics = false

 [anonymous-telemetry]
-enabled = false
\ No newline at end of file
+enabled = false
+
+[source]
+root_patterns = [
+  '/project',
+]
+
$ ./pants roots
15:43:33.92 [INFO] Initializing scheduler...
15:43:34.06 [INFO] Scheduler initialized.
project
And now things look better:
Copy code
$ ./pants package project:lambda
15:46:40.98 [INFO] Wrote dist/project/lambda.zip
    Runtime: python3.9
    Handler: lambdex_handler.handler
$ zipinfo dist/project/lambda.zip  | tail
-rw-r--r--  2.0 unx   106996 b- defN 80-Jan-01 00:00 .bootstrap/pex/venv/virtualenv_16.7.12_py
-rw-r--r--  2.0 unx      156 b- defN 80-Jan-01 00:00 .bootstrap/pex/version.py
-rw-r--r--  2.0 unx      533 b- defN 80-Jan-01 00:00 PEX-INFO
-rwxr-xr-x  2.0 unx     3524 b- defN 80-Jan-01 00:00 __main__.py
drwxr-xr-x  2.0 unx        0 b- defN 80-Jan-01 00:00 __pex__/
-rw-r--r--  2.0 unx     3498 b- defN 80-Jan-01 00:00 __pex__/__init__.py
-rw-r--r--  2.0 unx       60 b- defN 80-Jan-01 00:00 lambda_example.py
?rwxr-xr-x  2.0 unx       49 b- stor 80-Jan-01 00:00 LAMBDEX-INFO
?rwxr-xr-x  2.0 unx     1920 b- stor 80-Jan-01 00:00 lambdex_handler.py
133 files, 1749843 bytes uncompressed, 469435 bytes compressed:  73.2%
See here for more about source roots: https://www.pantsbuild.org/docs/source-roots#what-are-source-roots
s
Hey thanks for the response! Good catch on the source root, but I’m not sure that’s the issue. I updated the source root but still have the same problem when trying to run in Lambda. I’m packaging the docker file not the lambda zip, so 
./pants package project:my_image
I get a dockerfile out the other end, which I then upload to ECR and point my Lambda function at. The error after updating the source root is the same.
e
Have you poked in the image to confirm it has the new PEX without the project/ prefix on the embedded code?
That was certainly an issue.
s
Yeah let me try some local testing maybe it’s a caching thing in AWS
e
That said - how are you configuring the lambda on the AWS side? 
.app.lambdex_handler
implies AWS is looking at 
/
but your image seems to think 
WORKDIR /app
means something and perhaps it doesn't to lambda. I've not used lambdas with images, just with uploaded zips.
s
Oh okay the issue actually appears to be a Python 3.8 vs 3.9 issue, when I use 3.8 the code works fine and 3.9 it fails…
If I package it into a zip and create a 3.9 runtime Lambda function it works okay, it’s just if I use the Docker image
I can reproduce this locally by running 
docker run -p 9000:8080 myimage:latest
and then making the request with 
curl -XPOST "<http://localhost:9000/2015-03-31/functions/function/invocations>" -d '{}'
e
I see you've updated the example repo such that the lambda runtime(3.9) now matched the base image (3.9) and you've fixed up source roots. Are you saying that despite those fixes you still encounter issues using the image as your lambda deployment method?
s
Correct, you should be able to repro the issue locally using the current repo and the command above
e
Ok, yeah - reproed. So I just followed the directions here and things work: https://docs.aws.amazon.com/lambda/latest/dg/images-create.html
Copy code
jsirois@Gill-Windows:~/dev/nharada1/lambda-aws-pants-demo$ git diff
diff --git a/project/Dockerfile b/project/Dockerfile
index c658509..85a1639 100644
--- a/project/Dockerfile
+++ b/project/Dockerfile
@@ -3,6 +3,5 @@ FROM public.ecr.aws/lambda/python:3.9
 WORKDIR /build
 RUN yum install unzip -y
 COPY project/lambda.zip /build
-RUN unzip /build/lambda.zip -d /app
-WORKDIR /app
-CMD ["/app/lambdex_handler.handler"]
\ No newline at end of file
+RUN unzip /build/lambda.zip -d "${LAMBDA_TASK_ROOT}"
+CMD ["lambdex_handler.handler"]
Then:
Copy code
jsirois@Gill-Windows:~/dev/nharada1/lambda-aws-pants-demo$ ./pants package ::
10:23:49.38 [INFO] Completed: Building docker image myimage:latest
10:23:49.39 [INFO] Wrote dist/project/lambda.zip
    Runtime: python3.9
    Handler: lambdex_handler.handler
10:23:49.39 [INFO] Built docker image: myimage:latest
Docker image ID: e81026f23b35
$ docker run -p 9000:8080 myimage:latest
17 Sep 2022 17:20:46,509 [INFO] (rapid) exec '/var/runtime/bootstrap' (cwd=/build, handler=)
17 Sep 2022 17:20:54,838 [INFO] (rapid) extensionsDisabledByLayer(/opt/disable-extensions-jwigqn8j) -> stat /opt/disable-extensions-jwigqn8j: no such file or directory
17 Sep 2022 17:20:54,838 [WARNING] (rapid) Cannot list external agents error=open /opt/extensions: no such file or directory
START RequestId: 5e2942b6-f782-487b-9f33-2bb503a1a19e Version: $LATEST
Hello AWS!
END RequestId: 5e2942b6-f782-487b-9f33-2bb503a1a19e
REPORT RequestId: 5e2942b6-f782-487b-9f33-2bb503a1a19e  Init Duration: 0.18 ms  Duration: 233.78 ms     Billed Duration: 234 ms Memory Size: 3008 MB    Max Memory Used: 3008 MB
START RequestId: 2875ad8d-ebb6-473e-996d-de12f066f566 Version: $LATEST
Hello AWS!
END RequestId: 2875ad8d-ebb6-473e-996d-de12f066f566
REPORT RequestId: 2875ad8d-ebb6-473e-996d-de12f066f566  Duration: 1.45 ms       Billed Duration: 2 ms   Memory Size: 3008 MB    Max Memory Used: 3008 MB
^C17 Sep 2022 17:21:14,293 [INFO] (rapid) Received signal signal=interrupt
17 Sep 2022 17:21:14,293 [INFO] (rapid) Shutting down...
17 Sep 2022 17:21:14,293 [WARNING] (rapid) Reset initiated: SandboxTerminated
It seems to be the case you don't want to veer from LAMBDA_TASK_ROOT unless you really know what you're doing.
@straight-action-80318 where did you get inspiration from for your original problematic Dockerfile contents? Is there bad Pants docs somewhere training people to do this?
And, FWIW, with modern Pants / PEX you don't need lambdex any longer. You just need to use a magic root 
__pex__
package to have a PEX work like any other importable 
sys.path
entry. So this works too completely bypassing the lambdex handler:
Copy code
$ git diff
diff --git a/project/Dockerfile b/project/Dockerfile
index c658509..8f756ff 100644
--- a/project/Dockerfile
+++ b/project/Dockerfile
@@ -3,6 +3,5 @@ FROM public.ecr.aws/lambda/python:3.9
 WORKDIR /build
 RUN yum install unzip -y
 COPY project/lambda.zip /build
-RUN unzip /build/lambda.zip -d /app
-WORKDIR /app
-CMD ["/app/lambdex_handler.handler"]
\ No newline at end of file
+RUN unzip /build/lambda.zip -d "${LAMBDA_TASK_ROOT}"
+CMD ["__pex__.lambda_example.example_handler"]
In fact, you don't even need to unzip - this works:
Copy code
$ cat project/Dockerfile
FROM public.ecr.aws/lambda/python:3.9

COPY project/lambda.zip ${LAMBDA_TASK_ROOT}
ENV PYTHONPATH ${LAMBDA_TASK_ROOT}/lambda.zip
CMD ["__pex__.lambda_example.example_handler"]
s
Ahhh this does work, brilliant! Yeah, I got this code from the tutorial in the docs: https://www.pantsbuild.org/docs/awslambda-python
Is there a good way to introspect the 
__pex__
module? I’m now going to try and add another layer of folders, but simply doing 
CMD ["__pex__.inner_folder.lambda_example.example_handler"]
doesn’t work (
No module named '__pex__.inner_folder'"
) and I’m not sure how to view which methods pex exposes
e
The 
__pex__
psuedo-package exposes everything in the PEX including 3rdparty deps. What is most likely is there is an issue with your folder add. What does 
zipinfo your/pex
say? Or is this all committed to the example repo where I can look?
Ok, thanks for that doc link. Yeah - our docs are bad there. I wonder if they were ever good and something in AWS Lambda infra changed. I'll raise a question about that.
s
I think what happened is something changed with Python 3.9 and the docs got out of date? They were working for 3.8
And thank you! I’ve updated the repo with my folder example
Copy code
-rw-r--r--  2.0 unx      533 b- defN 80-Jan-01 00:00 PEX-INFO
-rwxr-xr-x  2.0 unx     3524 b- defN 80-Jan-01 00:00 __main__.py
drwxr-xr-x  2.0 unx        0 b- defN 80-Jan-01 00:00 __pex__/
-rw-r--r--  2.0 unx     3498 b- defN 80-Jan-01 00:00 __pex__/__init__.py
drwxr-xr-x  2.0 unx        0 b- defN 80-Jan-01 00:00 inner/
-rw-r--r--  2.0 unx       60 b- defN 80-Jan-01 00:00 inner/lambda_example.py
?rwxr-xr-x  2.0 unx       55 b- stor 80-Jan-01 00:00 LAMBDEX-INFO
?rwxr-xr-x  2.0 unx     1920 b- stor 80-Jan-01 00:00 lambdex_handler.py
This is what the zip looks like, excluding all the bootstrap stuff
e
Yeah, that looks like you found a bug in the 
__pex__
import magic. You need to add a dunder-init to 
inner
to get it to work. This works:
Copy code
jsirois@Gill-Windows:~/dev/nharada1/lambda-aws-pants-demo$ git reset --hard origin/HEAD
HEAD is now at b7137c0 Update for folder structure
jsirois@Gill-Windows:~/dev/nharada1/lambda-aws-pants-demo$ touch project/inner/__init__.py
jsirois@Gill-Windows:~/dev/nharada1/lambda-aws-pants-demo$ ./pants package ::
13:58:47.32 [INFO] Canceled: Building docker image myimage:latest
13:58:47.56 [INFO] Completed: Building docker image myimage:latest
13:58:47.56 [INFO] Wrote dist/project.inner/lambda.zip
    Runtime: python3.9
    Handler: lambdex_handler.handler
13:58:47.56 [INFO] Built docker image: myimage:latest
Docker image ID: fd7014075b07
jsirois@Gill-Windows:~/dev/nharada1/lambda-aws-pants-demo$ docker run -p 9000:8080 myimage:latest
17 Sep 2022 20:58:50,052 [INFO] (rapid) exec '/var/runtime/bootstrap' (cwd=/build, handler=)
17 Sep 2022 20:58:53,476 [INFO] (rapid) extensionsDisabledByLayer(/opt/disable-extensions-jwigqn8j) -> stat /opt/disable-extensions-jwigqn8j: no such file or directory
17 Sep 2022 20:58:53,476 [WARNING] (rapid) Cannot list external agents error=open /opt/extensions: no such file or directory
START RequestId: 9624f52c-9dea-4136-a75b-003f3b5c2288 Version: $LATEST
Hello AWS!
END RequestId: 9624f52c-9dea-4136-a75b-003f3b5c2288
REPORT RequestId: 9624f52c-9dea-4136-a75b-003f3b5c2288  Init Duration: 0.22 ms  Duration: 297.77 ms     Billed Duration: 298 ms Memory Size: 3008 MB    Max Memory Used: 3008 MB
^C17 Sep 2022 20:58:56,304 [INFO] (rapid) Received signal signal=interrupt
17 Sep 2022 20:58:56,304 [INFO] (rapid) Shutting down...
17 Sep 2022 20:58:56,304 [WARNING] (rapid) Reset initiated: SandboxTerminated
jsirois@Gill-Windows:~/dev/nharada1/lambda-aws-pants-demo$
s
Ahhhh amazing, thank you so much. Would not have gotten this working without your help John!
e
Thanks for the example repo - that makes debugging much easier.
12 Views
Open in Slack
PreviousNext
Powered by