Hello friends, <@U04SRQFFHKK> and I are doing some...
# general
Hello friends, @average-breakfast-91545 and I are doing some more macos + linux + docker + AWS lambda + pants shenanigans on the quest to get Carbon Re fully pantsy . We have got mostly everything working (bar Pytorch — expect more head-scratching questions in the coming weeks for your amusement here), but just writing here to get some feedback in case we are doing silly things (or rather, whether I am). Basic context: we have the majority of devs running macos machines (although some are using a couple of different linux distros), and we need to ship code onto AWS either in the form of Lambdas or ECS. This means building pex and shipping them in docker images (+ some mega exciting Lambda layer stuff that Bob is working on). So, in practice we currently have something along the lines of:
Copy code
        "FROM public.ecr.aws/lambda/python:3.9",
        "RUN yum -y update && yum -y install some list of build packages",
        ...  # Some unfun ENV shenanigans


    dependencies=["./entrypoint.py"],  # this brings all sorts of nasty python packages

        "FROM public.ecr.aws/lambda/python:3.9",
        "RUN yum -y update && yum -y install some runtime packages",
        ...  # more fun stuff
        "COPY path.to.package/entrypoint.pex /build/entrypoint.pex",
        "RUN unzip /build/entrypoint.pex -d \"${LAMBDA_TASK_ROOT}\"",
        "CMD [\"__pex__.path.to.package.entrypoint.handle\"]",
    ...  # some metadata for tags and stuff here
My current big question / worry is that I can’t seem to find a way of making
depend on
. Thus, I’m exploring a couple of options: 1. Run
pants package :aws_py39_pex_builder
followed by
pants package :lambda
, manually, wherever we need to. 2. Run a simple “check_and_build_maybe” function in
as a remediative step that carries out either
pants package :aws_py30_pex_builder
docker build …
(likely the latter). Am I missing a simpler way of solving this problem? Have I missed a glaringly obvious bit in the docs that would fix this problem for me? If not, and if I were to hypothetically look into adding a
field to
, how much trouble would I likely get myself into?
yeah, unfortunately a
can't be used as an environment yet: https://github.com/pantsbuild/pants/issues/17714 For 2, I do suspect that recursively calling
might fork-bomb your machine, so a direct
docker build
might be necessary. BTW, we also deploy lambdas (as zips) and to ECS, although we get away without running in environments by using complete platforms (all of our deps are either Python-only or provided as wheels, and you may not be so lucky!): https://github.com/pantsbuild/pants/discussions/18756 has a bunch of notes about it.
Oh man, these are great notes. Thank you! A preliminary comment:
• We’d prefer to deploy dependencies in a layer and then only our first-party source in the direct lambda package
Not sure how far you got on this, but you and @average-breakfast-91545 should chat 🙂
👍 1
For sure, I'm just about to start on https://github.com/pantsbuild/pants/issues/18879 using PEX's new
pex3 venv create --layout=flat-zipped
functionality, and then https://github.com/pantsbuild/pants/issues/18880 after that, as, hopefully, a small incremental step.
🚀 1
https://github.com/pantsbuild/pants/pull/19123 proposes an implementation of lambda layers. I'd been keen for your feedback @fast-belgium-77939 and @average-breakfast-91545, and how it connects to what you need. Thanks
🔥 1
I'll find some time to test it out. Thanks Huon! I'll find some time to try it out - thanks
👍 2
(I've merged that PR, so it'll hopefully come out in the next 2.18 dev release, which'll make it easier to test. Feedback is useful and important, can easily make lots of changes while it hasn't been in a stable release.)