#18756 Some recipes for AWS Lambda and Docker Python artefacts (including cross-building from macOS) New discussion created by huonw Questions often come up about building AWS Lambda and docker artefacts effectively on Slack. I've answered a few based on my experience using pants to deploy AWS Lambda (via zips) and Fargate for a while. This discussion post collects what is currently working for us to deploy dozens of lambdas and a handful of Fargate containers, to be a reference that I can link to. None of this is perfect, but it's been good enough, and, once we worked through the initial set-up, pants packaging has never been the cause of a deploy failure. We develop on ARM64 macOS, and deploy to Linux x86. We run pants natively, and try to do as much as work natively too as reasonable because that is noticeably more efficient than emulating the OS, and very much more efficient than emulating the architecture. We also implemented all this well before Pants 2.15 and its experimental environments feature (https://blog.pantsbuild.org/environments-simpler-multi-platform-workflows/): that can likely be used too, but I cannot report on that. Basic setup We're currently using Pants 2.14 and PEX v2.1.128. Upgrading the latter is required to pick up a few bug fixes that were affecting our Lambda and docker builds. Our

pants.toml

looks something like:

[GLOBAL]
pants_version = "2.14.0"

backend_packages = [
  "pants.backend.python",
  "pants.backend.awslambda.python",
  "pants.backend.docker",
  ...
]

...

[pex-cli]
version = "v2.1.128"
known_versions = [
    "v2.1.128|macos_arm64|a48a461a9e9f490476aa75976dbe58d3263691a8895a53d11fdaaf8a3bc223a5|4081603",
    "v2.1.128|macos_x86_64|a48a461a9e9f490476aa75976dbe58d3263691a8895a53d11fdaaf8a3bc223a5|4081603",
    "v2.1.128|linux_x86_64|a48a461a9e9f490476aa75976dbe58d3263691a8895a53d11fdaaf8a3bc223a5|4081603",
    "v2.1.128|linux_arm64|a48a461a9e9f490476aa75976dbe58d3263691a8895a53d11fdaaf8a3bc223a5|4081603"
]

...

Python AWS Lambda zip artefacts Most important: use the

complete_platforms=

parameter, NOT

runtime=

• We build dozens of lambdas via the built-in

python_awslambda

target: https://www.pantsbuild.org/docs/awslambda-python • The most important part of getting this working reliably is using a 'complete platform' JSON file instead of the

runtime

parameter, to tell PEX about the environment: • This allows pants/PEX to select the appropriate dependencies, even when building on a different platform (like macOS vs. Linux). This does require that dependencies are available as a wheel (bdist) and/or are pure Python, because platform-specific dependencies cannot be cross-built. This has worked fine for us in practice, with the dependencies we use. • We've generated an appropriate file (many months ago) by running the following in a Lambda (in retrospect, I imagine it could've been run in a one of the AWS-provided docker images, but... this post is just describing what is working for us):

import subprocess
        
        def lambda_handler(event, context):
            subprocess.run(
                """
                pip install --target=/tmp/subdir pex
                PYTHONPATH=/tmp/subdir /tmp/subdir/bin/pex3 interpreter inspect --markers --tags
                """,
                shell=True
            )
            return {'statusCode': 200, 'body': "{}"}

• The output is put into

3rdparty/platforms/aws_lambda_python_3_9.json

with a

BUILD

file:

# 3rdparty/platforms/BUILD
        file(
            name="aws_lambda_python_3_9",
            source="aws_lambda_python_3_9.json",
        )

• For convenience, here's the literal file we're using (NB. this is Python 3.9 in

ap-southeast-2

, as of 8 months ago, so your environment may differ): lambda-complete-platform.zip • To actually build the lambdas we have macro:

def your_name_here_python_awslambda(*, **kwargs):
        kwargs.setdefault("dependencies", []).extend(
            [
                # FIXME <https://github.com/pantsbuild/pants/issues/15454>
                "!!path/to/requirements#mypy",
                "!!path/to/requirements#sqlalchemy-stubs",
            ]
        )
        kwargs["complete_platforms"] = ["3rdparty/platforms:aws_lambda_python_3_9"]  
        python_awslambda(**kwargs)

• The exclusions are working around #15454, to reduce the size of the deployed package. You may need to adjust them, based on what packages you use. • This can be called like the normal

python_awslambda

target, and generates the same sort of output (a zip in

dist/

): (As above, note that the

runtime=

parameter is not used.)

# some/code/BUILD
    your_name_here_python_awslambda(
        name="some_name",
        handler="path/to/file.py:your_handler",
    )

• We deploy these via CDK, outside of pants, using a construct that takes constructs the appropriate

dist/some.code/some_name.zip

path to pass to `lambda.Code.fromAsset`:

pants package ::
    cdk deploy path/to/stack.js

• Things we'd like to improve, eventually: • We'd prefer to deploy dependencies in a layer and then only our first-party source in the direct lambda package • #15454 • Using the fact that the target environment is static to layout an appropriate .zip file, and avoid the lambdex/PEX dynamic environment detection at start up (this adds about 1s to the cold start, it seems) • Improving build-time/cacheability: the current all-in-one zip files means that a minor first-party code that affects a bunch of lambdas change might have to generate a fresh 300MB of .zip, which both takes time and a lot of cache space. The first point above likely improves this: splitting 1st and 3rd party packages into separate packages. Docker image artefacts from PEX • We build docker images inspired by https://blog.pantsbuild.org/optimizing-python-docker-deploys-using-pants/ • We only build them via pants, and do not push them, and instead re-tag them outside pants before pushing, as we haven't had time/strong-need to work through parametrising different registries for prod vs. development images. • Again, we use

complete_platforms

to be able to build PEX files natively on macOS: • we currently run things in

python:3.9-bullseye

containers, and thus have added

docker_python_3_9_bullseye_arm64.json

and

docker_python_3_9_bullseye_amd64.json

files to `3rdparty/platforms`; here's the

BUILD

file including how we generate them:

# docker run -it --platform linux/arm64 python:3.9.10-bullseye bash -c 'pip install pex; pex3 interpreter inspect --markers --tags'
        # docker run -it --platform linux/amd64 python:3.9.10-bullseye bash -c 'pip install pex; pex3 interpreter inspect --markers --tags'
        files(
            name="docker_python_3_9_bullseye",
            sources=["docker_python_3_9_bullseye_*.json"],
        )

• We run Linux/arm64 images on dev machines and Linux/amd64 images on CI/in production, and thus include complete platforms for both (running amd64 images on arm64 dev machines is very slow) • To actually build images we have a macro: ``` BASE_IMAGE = "python:3.9-bullseye" COMPLETE_PLATFORMS = ("3rdparty/platforms:docker_python_3_9_bullseye",) def xo_docker_image_from_pex( *, name, pex_entry_point, extra_instructions, pex_dependencies=None, **kwargs, ): """ Build a docker image using PEX… pantsbuild/pants