Hi, is there a target to export the python venv folder, created by

pants export ...

and stored in

dist/export/python/virtualenvs/...

? I’d like to make the export of the venv a dependancy of the creation of the docker image, which needs to copy the python interpreter. I was thinking about doing something like this, but pants can’t run another pants command :

run_shell_command(
    name="generate_venv",
    command="pants export --export-resolve='[\"python-default\"]'",
    workdir="/",
)
docker_image(
    name="docker_im",
    dependencies=["src/repo:generate_venv"],
)

$ pants run src/repo:generate_venv
Another pants invocation is running. Will wait up to 60.0 seconds for it to finish before giving up.

If I understand correctly, Pants team recommends using pex files for populating docker. What's the use case specifically for exported venv?

Take a look at the blog https://www.pantsbuild.org/blog/2022/08/02/optimizing-python-docker-deploys-using-pants

Yeah, creating venvs from pexes is the “blessed” way of getting a deployable venv out

I’d like to use the docker container in CI and interactive mode, for development. So i want to be able to use

pip install

for example, and not have to re-export everytime i want to use a new package. I don’t know if PEX files are flexible enough for this

You can pip install into unpacked pex

When you use

pants export

, you get a full usable venv. I tried to use a PEX binary target to export my lockfile to a venv the same way as pants export, but i can’t manage to get it the same way

pex_binary(
    name="venv",
    layout='loose',
    dependencies=[":python_requirements"],
    resolve="python-default",
    venv_site_packages_copies=True,
)

This is what i get with pex_binary :

$ tree -a -L 3 dist/src.ml/ml-venv.pex/
dist/src.ml/ml-venv.pex/
├── .bootstrap
│   └── pex
│       ├── argparse.py
│       ├── asserts.py
│       ├── atomic_directory.py
│       ├── attrs.py
│       ├── auth.py
│       ├── bootstrap.py
│       ├── commands
│       ├── common.py
│       ├── compatibility.py
│       ├── compiler.py
│       ├── dependency_manager.py
│       ├── dist_metadata.py
│       ├── enum.py
│       ├── environment.py
│       ├── exclude_configuration.py
│       ├── executor.py
│       ├── fetcher.py
│       ├── finders.py
│       ├── fingerprinted_distribution.py
│       ├── hashing.py
│       ├── inherit_path.py
│       ├── __init__.py
│       ├── interpreter_constraints.py
│       ├── interpreter.py
│       ├── jobs.py
│       ├── layout.py
│       ├── __main__.py
│       ├── network_configuration.py
│       ├── orderedset.py
│       ├── pep_376.py
│       ├── pep_425.py
│       ├── pep_427.py
│       ├── pep_440.py
│       ├── pep_503.py
│       ├── pep_508.py
│       ├── pex_bootstrapper.py
│       ├── pex_builder.py
│       ├── pex_info.py
│       ├── pex.py
│       ├── pex_warnings.py
│       ├── platforms.py
│       ├── pth.py
│       ├── pyenv.py
│       ├── rank.py
│       ├── requirements.py
│       ├── resolver.py
│       ├── result.py
│       ├── sh_boot.py
│       ├── sorted_tuple.py
│       ├── targets.py
│       ├── third_party
│       ├── tools
│       ├── tracer.py
│       ├── typing.py
│       ├── util.py
│       ├── variables.py
│       ├── vendor
│       ├── venv
│       ├── version.py
│       ├── wheel.py
│       └── ziputils.py
├── .deps
│   ├── iniconfig-2.0.0-py3-none-any.whl
│   │   ├── iniconfig
│   │   ├── iniconfig-2.0.0.dist-info
│   │   └── .layout.json
│   ├── numpy-2.0.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
│   │   ├── .layout.json
│   │   ├── numpy
│   │   ├── numpy-2.0.1.dist-info
│   │   ├── numpy.libs
│   │   └── .prefix
│   ├── packaging-24.1-py3-none-any.whl
│   │   ├── .layout.json
│   │   ├── packaging
│   │   └── packaging-24.1.dist-info
│   ├── pluggy-1.5.0-py3-none-any.whl
│   │   ├── .layout.json
│   │   ├── pluggy
│   │   └── pluggy-1.5.0.dist-info
│   └── pytest-8.3.2-py3-none-any.whl
│       ├── .layout.json
│       ├── .prefix
│       ├── py.py
│       ├── _pytest
│       ├── pytest
│       └── pytest-8.3.2.dist-info
├── __main__.py
├── __pex__
│   └── __init__.py
└── PEX-INFO

And this is what i expected to get instead, a virtual environment file structure :

$ tree -a -L 2 dist/export/python/virtualenvs/python-default/3.12.4/
dist/export/python/virtualenvs/python-default/3.12.4/
├── bin
│   ├── activate
│   ├── activate.csh
│   ├── activate.fish
│   ├── Activate.ps1
│   ├── dotenv
│   ├── editorconfig
│   ├── f2py
│   ├── faker
│   ├── ff
│   ├── flask
│   ├── fonttools
│   ├── functions-framework
│   ├── functions_framework
│   ├── functions-framework-python
│   ├── functions_framework_python
│   ├── gunicorn
│   ├── httpx
│   ├── js-beautify
│   ├── litestar
│   ├── markdown-it
│   ├── normalizer
│   ├── pip
│   ├── pip3
│   ├── pip3.12
│   ├── pyftmerge
│   ├── pyftsubset
│   ├── pygmentize
│   ├── pyrsa-decrypt
│   ├── pyrsa-encrypt
│   ├── pyrsa-keygen
│   ├── pyrsa-priv2pub
│   ├── pyrsa-sign
│   ├── pyrsa-verify
│   ├── py.test
│   ├── pytest
│   ├── python -> python3.12
│   ├── python3 -> python3.12
│   ├── python3.12 -> /home/gregoire/.asdf/installs/python/3.12.4/bin/python3.12
│   ├── rich-click
│   ├── tb-gcp-uploader
│   ├── tqdm
│   ├── ttx
│   ├── uvicorn
│   ├── watchfiles
│   └── watchmedo
├── include
│   └── python3.12
├── lib
│   └── python3.12
├── lib64 -> lib
├── __main__.py
├── pex -> __main__.py
├── PEX-INFO
├── .pex-venv-scope
├── pyvenv.cfg
└── share
    └── man

Did you read the blog post?

Yes, am i missing something ?

well, there is a step to convert it from pex to venv

"RUN PEX_TOOLS=1 /usr/local/bin/python3.10 /binary.pex venv --scope=deps --compile /bin/app",

Oh ok thanks 🙌

anyway > But if you want your development in a container, then I suggest installing pants inside the container and mounting your whole repo in it. Then you can do pants export from inside the container

I’d like to avoid having the steps of 1. Start the docker, mount the repo 2. Build the python interpreter -> This take some time Ideally when they run the container, it’s already working quickly

why do you want your development in the container in the first place?

To make it fast to setup again

what's slow in pants setup?

I guess it could be fast if the dependancies are already there inside the docker image, so you don’t have to download them again. Also you have the requirement of always mounting the codebase, ideally in the same spot, but it’s not a big requirement. The main idea is that you change dependancies / download a new docker image rarely, but you can start a docker container often. So it’s nice if the dependancies are already cached somewhere

well, pants uses pip cache to cache the dependencies, so

pants export

is slow only the first time. But even simpler - when you have the venv, isn't it ready to use any time after that? why do you need the image?

The image is meant to be optional, to have a reliable way to have something that works all the time. But you can install without. I will try to install pants inside the docker image and fill up the cache, that might be a good way

the thing is pants is designed to be reproducible, so that you don't need an image

It’s always better if you can have a Docker container anyone can use, and that you can use when you rent a VM in the cloud, or things like that. I managed to make it work by, in the dockerfile :

1 - install pants
2 - pull my repo
3 - generate the python environment with pants export, which fills up the pants cache
4 - delete the repo, but keep the Pants cache

So now the cache has all the dependancies needed to regenerate the interpreter fast, when you run

docker run

Thanks for your help @square-psychiatrist-19087

👍