https://pantsbuild.org/ logo
#general
Title
# general
f

full-toothbrush-75676

04/04/2022, 4:17 PM
Hi Pants! I want to generate a “runtime” Docker image that contains the 1P and 3P dependencies for a subset of my codebase, but does not use a
pex_binary
as an entry point. I want to be able to import and use these 1P and 3P dependencies in code that is run inside of the Docker image. What is the “right way” to do this? I can think of a couple of options: 1. Make a
python_distribution
, then
RUN pip install …
it in my
docker_image
. a. This probably won’t let me take advantage of Pants’ caching for the 3P dependencies 😕, is that right? 2. Make a
pex_binary
and use some fancy logic to “expand” the pex’s
venv
in the
docker_image
to make my 1P and 3P code importable a. It looks like I can build the pex in “venv” mode and then activate the venv. That works great in isolation, however, I need this to work inside of an active conda environment in the container, which doesn’t work 😅 (kind of strange i know I know haha) Thanks in advance and let me know if this question is unclear.
e

enough-analyst-54434

04/04/2022, 4:24 PM
One way is to use a
pex_binary
with no
entry_point
or
script
. Make sure your
pex_binary
has either
execution_mode=venv
or
include_tools=True
(or both). Then install your PEX like so: https://pex.readthedocs.io/en/v2.1.75-public-signed/recipes.html#pex-app-in-a-container. Finally, use that venv you installed like any other.
There are other benefits to this way of doing it outlined in that Pex documentation.
i

icy-hair-30586

04/05/2022, 4:32 PM
Can i install the dependencies of a pex into the global environment in a docker image? Some other tools in the base image expect to not use venvs, so it would make things more convenient to skip specifying and creating a venv
1
👍 1
e

enough-analyst-54434

04/05/2022, 4:58 PM
Not in any clean way today. That would be a new feature. The
--scope deps
and
--scope srcs
options highlighted here 1st create a new venv and then add sources or deps to it. There would have to be a mode where they don't 1st create a new venv, but are just pointed at an existing one. Something like
PEX_TOOLS=1 /usr/local/bin/python3.10 /my-app.pex venv --update --scope=deps --compile /usr/local
. That example updates the
/usr/local
system installed Python 3.10 in this case (i.e.: it would populate
/usr/local/lib/python3.10/site-packages
). If you have a need for that sort of feature, definitely file a Pex issue spelling all this out. If you want to try implementing the feature, I'm happy to help guide you.
Well, I guess you could actually try this @icy-hair-30586 :
Copy code
PEX_TOOLS=1 /usr/local/bin/python3.10 /my-app.pex repository extract --dest-dir /tmp/pex-deps-dump-wheels
/usr/local/bin/pip3.10 install --no-deps /tmp/pex-deps-dump-wheels/*.whl
That would get all the third party dependencies in your PEX file appropriate to the system Python 3.10 interpreter at
/usr/local/bin/python3.10
extracted as wheels and then installed in the site-packages of that system
/usr/local/bin/python3.10
interpreter.
🙌 3
This is generally a very very bad idea for a system interpreter, but in a container image, it actually is harmless to try.
i

icy-hair-30586

04/07/2022, 7:04 AM
Now if this supported the
--scope=srcs/deps
like the
venv
command that would be perfect :)
3 Views