https://pantsbuild.org/ logo
#general
Title
# general
f

fast-belgium-77939

05/10/2023, 10:10 PM
Hello friends, @average-breakfast-91545 and I are doing some more macos + linux + docker + AWS lambda + pants shenanigans on the quest to get Carbon Re fully pantsy . We have got mostly everything working (bar Pytorch — expect more head-scratching questions in the coming weeks for your amusement here), but just writing here to get some feedback in case we are doing silly things (or rather, whether I am). Basic context: we have the majority of devs running macos machines (although some are using a couple of different linux distros), and we need to ship code onto AWS either in the form of Lambdas or ECS. This means building pex and shipping them in docker images (+ some mega exciting Lambda layer stuff that Bob is working on). So, in practice we currently have something along the lines of:
Copy code
docker_image(
    name="aws_py39_pex_builder",
    instructions=[
        "FROM public.ecr.aws/lambda/python:3.9",
        "RUN yum -y update && yum -y install some list of build packages",
        ...  # Some unfun ENV shenanigans
    ],
)

docker_environment(
    name="aws_lambda_py39",
    image="aws_py39_pex_builder",
)

pex_binary(
    name="entrypoint",
    dependencies=["./entrypoint.py"],  # this brings all sorts of nasty python packages
    environment="aws_lambda_py39",
)

docker_image(
    name="lambda",
    dependencies=[":entrypoint"],
    instructions=[
        "FROM public.ecr.aws/lambda/python:3.9",
        "RUN yum -y update && yum -y install some runtime packages",
        ...  # more fun stuff
        "COPY path.to.package/entrypoint.pex /build/entrypoint.pex",
        "RUN unzip /build/entrypoint.pex -d \"${LAMBDA_TASK_ROOT}\"",
        "CMD [\"__pex__.path.to.package.entrypoint.handle\"]",
    ],
    ...  # some metadata for tags and stuff here
)
My current big question / worry is that I can’t seem to find a way of making
aws_lambda_py39
depend on
aws_py39_pex_builder
. Thus, I’m exploring a couple of options: 1. Run
pants package :aws_py39_pex_builder
followed by
pants package :lambda
, manually, wherever we need to. 2. Run a simple “check_and_build_maybe” function in
.pants.bootstrap
as a remediative step that carries out either
pants package :aws_py30_pex_builder
or
docker build …
(likely the latter). Am I missing a simpler way of solving this problem? Have I missed a glaringly obvious bit in the docs that would fix this problem for me? If not, and if I were to hypothetically look into adding a
dependencies
field to
docker_environment
, how much trouble would I likely get myself into?
b

broad-processor-92400

05/10/2023, 10:17 PM
yeah, unfortunately a
docker_image
can't be used as an environment yet: https://github.com/pantsbuild/pants/issues/17714 For 2, I do suspect that recursively calling
pants
in
.pants.bootstrap
might fork-bomb your machine, so a direct
docker build
might be necessary. BTW, we also deploy lambdas (as zips) and to ECS, although we get away without running in environments by using complete platforms (all of our deps are either Python-only or provided as wheels, and you may not be so lucky!): https://github.com/pantsbuild/pants/discussions/18756 has a bunch of notes about it.
f

fast-belgium-77939

05/10/2023, 10:24 PM
Oh man, these are great notes. Thank you! A preliminary comment:
• We’d prefer to deploy dependencies in a layer and then only our first-party source in the direct lambda package
Not sure how far you got on this, but you and @average-breakfast-91545 should chat 🙂
👍 1
b

broad-processor-92400

05/10/2023, 10:27 PM
For sure, I'm just about to start on https://github.com/pantsbuild/pants/issues/18879 using PEX's new
pex3 venv create --layout=flat-zipped
functionality, and then https://github.com/pantsbuild/pants/issues/18880 after that, as, hopefully, a small incremental step.
🚀 1
https://github.com/pantsbuild/pants/pull/19123 proposes an implementation of lambda layers. I'd been keen for your feedback @fast-belgium-77939 and @average-breakfast-91545, and how it connects to what you need. Thanks
🔥 1
a

average-breakfast-91545

05/24/2023, 8:50 AM
I'll find some time to test it out. Thanks Huon! I'll find some time to try it out - thanks
👍 2
b

broad-processor-92400

05/30/2023, 5:50 AM
(I've merged that PR, so it'll hopefully come out in the next 2.18 dev release, which'll make it easier to test. Feedback is useful and important, can easily make lots of changes while it hasn't been in a stable release.)
5 Views