pybind11 + pex + docker query:
# general
w
pybind11 + pex + docker query:
We've enjoyed success using the recipe outlined in https://blog.pantsbuild.org/optimizing-python-docker-deploys-using-pants/ , it's been great. However, we have some modules implemented using pybind11 (and more specifically cppimport). Compiling these sources at the end of the docker stages feel icky:
Copy code
FROM python:3.10-slim
COPY --from=dependencies /app /app
COPY --from=source /app /app

RUN buildDeps='build-essential' && runtimeDeps='libgomp1' \
    && apt-get update \
    && apt-get install --no-install-recommends -y $buildDeps $runtimeDeps \
    && python -m cppimport build \
    && apt-get purge -y --auto-remove $buildDeps \
    && rm -rf /var/lib/apt/lists/*
because these sources changes rarely, but now any source code change forces a rebuild of the c++ modules.
Is there an obvious way to 1. keep the awesome pants inference 2. Run the RUN instructions in a separate build (perhaps an additional "pex layer"?)
b
I actually have us compiling our py module using pybind11 using a hacked-together plugin (to compile the module) on the dev boxes. I use the http_source source of the file target to download pybind11
That being said, yeah it shouldn't be hard to keep tacking on more docker build stages, as long as you copy it into the "right" place in the final image. Search the pants site for transitive excludes. Use that on the PEX that depends on the compiled module to omit that dependency
✔️ 1