Hello friends, @average-breakfast-91545 and I are doing some more macos + linux + docker + AWS lambda + pants shenanigans on the quest to get Carbon Re fully ✨ pantsy ✨. We have got mostly everything working (bar Pytorch — expect more head-scratching questions in the coming weeks for your amusement here), but just writing here to get some feedback in case we are doing silly things (or rather, whether I am). Basic context: we have the majority of devs running macos machines (although some are using a couple of different linux distros), and we need to ship code onto AWS either in the form of Lambdas or ECS. This means building pex and shipping them in docker images (+ some mega exciting Lambda layer stuff that Bob is working on). So, in practice we currently have something along the lines of:

docker_image(
    name="aws_py39_pex_builder",
    instructions=[
        "FROM public.ecr.aws/lambda/python:3.9",
        "RUN yum -y update && yum -y install some list of build packages",
        ...  # Some unfun ENV shenanigans
    ],
)

docker_environment(
    name="aws_lambda_py39",
    image="aws_py39_pex_builder",
)

pex_binary(
    name="entrypoint",
    dependencies=["./entrypoint.py"],  # this brings all sorts of nasty python packages
    environment="aws_lambda_py39",
)

docker_image(
    name="lambda",
    dependencies=[":entrypoint"],
    instructions=[
        "FROM public.ecr.aws/lambda/python:3.9",
        "RUN yum -y update && yum -y install some runtime packages",
        ...  # more fun stuff
        "COPY path.to.package/entrypoint.pex /build/entrypoint.pex",
        "RUN unzip /build/entrypoint.pex -d \"${LAMBDA_TASK_ROOT}\"",
        "CMD [\"__pex__.path.to.package.entrypoint.handle\"]",
    ],
    ...  # some metadata for tags and stuff here
)

My current big question / worry is that I can’t seem to find a way of making

aws_lambda_py39

depend on

aws_py39_pex_builder

. Thus, I’m exploring a couple of options: 1. Run

pants package :aws_py39_pex_builder

followed by

pants package :lambda

, manually, wherever we need to. 2. Run a simple “check_and_build_maybe” function in

.pants.bootstrap

as a remediative step that carries out either

pants package :aws_py30_pex_builder

or

docker build …

(likely the latter). Am I missing a simpler way of solving this problem? Have I missed a glaringly obvious bit in the docs that would fix this problem for me? If not, and if I were to hypothetically look into adding a

dependencies

field to

docker_environment

, how much trouble would I likely get myself into?

yeah, unfortunately a

docker_image

can't be used as an environment yet: https://github.com/pantsbuild/pants/issues/17714 For 2, I do suspect that recursively calling

pants

in

.pants.bootstrap

might fork-bomb your machine, so a direct

docker build

might be necessary. BTW, we also deploy lambdas (as zips) and to ECS, although we get away without running in environments by using complete platforms (all of our deps are either Python-only or provided as wheels, and you may not be so lucky!): https://github.com/pantsbuild/pants/discussions/18756 has a bunch of notes about it.

Oh man, these are great notes. Thank you! A preliminary comment:

• We’d prefer to deploy dependencies in a layer and then only our first-party source in the direct lambda package

Not sure how far you got on this, but you and @average-breakfast-91545 should chat 🙂

For sure, I'm just about to start on https://github.com/pantsbuild/pants/issues/18879 using PEX's new

pex3 venv create --layout=flat-zipped

functionality, and then https://github.com/pantsbuild/pants/issues/18880 after that, as, hopefully, a small incremental step.

I'll find some time to test it out. Thanks Huon! I'll find some time to try it out - thanks

(I've merged that PR, so it'll hopefully come out in the next 2.18 dev release, which'll make it easier to test. Feedback is useful and important, can easily make lots of changes while it hasn't been in a stable release.)