https://pantsbuild.org/ logo
Join Slack
Powered by
Is there a way to build a docker-image that doesn'...
# general
r
Is there a way to build a docker-image that doesn't use a PEX file to execute? I'm running into more issues (this time with pex and spawned process) and wondering if we can just avoid pex altogether. How can I use pants to pull in all requirements for a target and bundle it into a docker image that can be run as a traditional 
requirements.txt + source files
?
b
r
Thanks! I'm messing around with this, it looks like it's just a wheel that (i guess) we can COPY into the image and then pip install. I wish it would generate pinned dependencies of everything though
b
write a script to parse the output of generate-lockfiles?
g
You could build a pex with only the deps, and run your code against that venv.
e.g.
Copy code
$ echo "import numpy as np; print(np.array([1, 2, 3]).sum())" > script.py
$ pex -o np.pex numpy
$ ./np.pex script.py
6
In pants terms, you can use 
include_sources=False
on a pex_binary to only package deps for the entrypoint. Feasibly you could then mount the whole src tree and run against, and as long as your deps don't change it should work.
👀 1
If docker isn't a requirement for dev, there's 
--loop
as well.
p
to elaborate on Tom's suggestion, this is the pattern we follow to build separate deps/src pexes and then incremental builds only need to build the src pex: https://www.pantsbuild.org/blog/2022/08/02/optimizing-python-docker-deploys-using-pants
👍 1
r
Thanks @powerful-scooter-95162, we're currently going with that approach but running into a couple of issues (the two I recently posted about in #C046T6T9U @gorgeous-winter-99296 what you described is I tihnk what we want, is it something like:
Copy code
pex_binary(name="deps", layout="packed", include_sources=False, ...)
python_distribution(name="srcs", ...)
docker_image(depends=[":deps", ":srcs"]
And then in Dockerfile
Copy code
COPY ./deps.pex ./deps.pex
COPY ./srcs.whl ./srcs.whl
CMD [
  "source", "./deps.pex/venv/bin/activate",
  "&&",
  "pip install -e ./srcs.whl",
  "&&",
  "my_entrypoint",
]
It looks like I can just install all dependencies from 
deps.pex/.deps
to get a bunch of wheels
actualy all the files are here in the pex; maybe I can just hack in my own virtualenv based on these dependencies. I need a deeper understanding of how pex works, wonder if it's something trivial
Ok I got this working, the subprocess issue is gone now that I'm not using PEX to run it. For future reference, here's what I did. It'd be nice to either document and/or make this easier to do: 1. Follow the 2-pex multi-layer approach with 
serve-deps.pex
and 
serve-srcs.pex
2. In Dockerfile, for serve-deps, stick with the recipe in the blog post that @powerful-scooter-95162 mentioned 3. For 
serve-srcs.pex
just run COPY instead of using pextools. This will pull over the sources, ie:
Copy code
RUN mkdir -p /bin/app/
RUN cp -r /serve-srcs.pex/* /bin/app/
RUN cp -r /serve-srcs.pex/.[^.]* /bin/app/
4. Use a shell script to activate the virtualenv and run your start script, eg
Copy code
#! /bin/bash

echo "Sourcing /bin/app/bin/activate"
. /bin/app/bin/activate
export PYTHONPATH=/bin/app

python my_app/my_serve.py
I'm almost certain this will unlock the reloadable server as well. Thanks everyone for weighing in, really awesome community here 🙂
s
Thanks for sharing the solution! Been interested in something like this myself
❤️ 1
21 Views
Open in Slack
PreviousNext
Powered by