https://pantsbuild.org/ logo
#general
Title
# general
b

brave-hair-402

02/17/2023, 6:44 PM
Has anyone example of a docker file build for AWS lambda with a container image python 3.10 (or 3.11)? (no official support yet https://github.com/aws/aws-lambda-base-images/issues/31) I have done this successfully using poetry packages and awslambdaric, but I am having some issues getting this working with PEX files.
e

enough-analyst-54434

02/17/2023, 6:47 PM
I don't, but what does the Poetry Dockerfile look like? Seeing that I'm sure I can point out how you do this with a PEX.
b

brave-hair-402

02/17/2023, 6:52 PM
My poetry (pre-pants) version looks something like this.
Copy code
# Define global args
# ARG FUNCTION_DIR="/app"
ARG BUILD_DIR="/build"
ARG RUNTIME_VERSION="3.10"
ARG DISTRO_VERSION="slim-bullseye"

FROM public.ecr.aws/docker/library/python:${RUNTIME_VERSION}-${DISTRO_VERSION} as base-image

ENV PYTHONFAULTHANDLER=1 \
    PYTHONHASHSEED=42 \
    PYTHONUNBUFFERED=1 \
    POETRY_VERSION=1.1.14
# Stage 2 - build function and dependencies
FROM base-image AS build-image
# Install aws-lambda-cpp build dependencies
RUN apt-get update && \
  apt-get install -y \
  g++ \
  make \
  cmake \
  unzip \
  libcurl4-openssl-dev

WORKDIR /app
RUN pip install "poetry==$POETRY_VERSION" && python -m venv /build/venv

COPY . .
RUN . /build/venv/bin/activate && cd projects/some_service/lambda_function && poetry install --no-dev --no-root
COPY projects/some_service/lambda_function/app.py /build/

FROM base-image
WORKDIR /app
COPY --from=build-image build /app

ENTRYPOINT [ "/app/venv/bin/python", "-m", "awslambdaric" ]
CMD [ "app.handler" ]
e

enough-analyst-54434

02/17/2023, 6:57 PM
So, with Pants (
pex_binary(include_tools=True, complete_platforms=[...])
/ Pex (
--complete-platform=... --include-tools
), it would be:
Copy code
COPY dist/my.pex /my.pex
RUN PEX_TOOLS=1 /my.pex venv /build/venv
Instead of the
RUN pip install ...
and `RUN . /build/venv/bin/activate ...`lines; otherwise the same.
🙌 1
The trick is to produce a complete platform JSON file. This is a 1-time step per image and allows Pants to build a PEX for AWS Linux on your machine. The
complete_platforms
field has docs on this that you'll need to follow over to Pex docs, but I can also give you a 1-liner if that's still unclear. Alternatively, in Pants 2.15.x, there is support for performing certain build actions in containers. One of those is building a PEX. That is described here: https://www.pantsbuild.org/v2.15/docs/environments This latter environments option is experimental, but the most solid approach.
You could obviously also do it the exact same way you do with poetry. Install pants in the container, run pants to get a PEX, then use the PEX_TOOLS trick.
That saves you complete_platforms / environments experimental feature "cross-building". Poetry, in contrast, doesn't support cross-building at all IIUC, thus the need to install and run it temporarily in the container.
@brave-hair-402 so 3 options, 1 being do it just like you do it with poetry, the other 2 being "cross-build" techniques that actually build the artifact outside of your final Docker image.
Make some sort of sense?
The only hard bit here is the potential cross-build depending on which option you choose. Once you have a PEX compatible with the Linux target platform though, AWS Lambdas aside, the general PEX -> container advice is documented here: + https://pex.readthedocs.io/en/v2.1.122/recipes.html#pex-app-in-a-container + https://blog.pantsbuild.org/optimizing-python-docker-deploys-using-pants/
b

brave-hair-402

02/17/2023, 7:26 PM
This makes sense. I have to read these docs a bit better to evaluate what my best choice is; but using the 2.15 environments to do cross-builds looks promising. I will try both options and see if I run into any issues (after reading the docs). Thank you.
I have now got the lambda build working, ignoring cross-builds (I am building on a platform compatible with the images so it works). I am now trying to get he environment working with the following BUILD file
Copy code
docker_image(name="docker_image", dependencies=[":bin"])

python_sources()

docker_environment(
  name="python_bullseye",
  platform="linux_x86_64",
  image="python:3.10-slim-buster",
)

python_requirement(
    name="awslambdaric",
    requirements=["awslambdaric==2.0.4"],
)

pex_binary(
    name="bin",
    entry_point="app.py",
    include_tools=True,
    dependencies=[":awslambdaric"],
    environment="python_bullseye"
)
But I get the following error: `UnrecognizedEnvironmentError: Unrecognized environment name
python_bullseye
from the
environment
field from the target`
This is what I used to get a custom python 3.10 image working with a pex file.
Copy code
# Define custom function directory
ARG FUNCTION_DIR="/var/task/"

FROM python:3.10-slim-buster as build-image

# Include global arg in this stage of the build
ARG FUNCTION_DIR
RUN mkdir -p ${FUNCTION_DIR}

# Install aws-lambda-cpp build dependencies
RUN apt-get update && \
  apt-get install -y \
  g++ \
  make \
  cmake \
  unzip \
  libcurl4-openssl-dev

# Install the function's dependencies

COPY projects.web_service.binary_name/bin.pex .
RUN PEX_TOOLS=1 ./bin.pex venv ${FUNCTION_DIR}
RUN cp -r ${FUNCTION_DIR}/lib/python3.10/site-packages/binary_name/* ${FUNCTION_DIR}

FROM python:3.10-slim-buster

# Include global arg in this stage of the build
ARG FUNCTION_DIR
# Set working directory to function root directory
WORKDIR ${FUNCTION_DIR}

# Copy in the built dependencies
COPY --from=build-image ${FUNCTION_DIR} ${FUNCTION_DIR}
RUN chmod -R 0755 .

ENTRYPOINT [ "/var/task/bin/python", "-m", "awslambdaric" ]
CMD ["app.handler"]
22 Views