abundant-hospital-56388
07/21/2022, 8:31 AMexecution_mode = "venv"
for the target.
In particular I get the error ImportError: Failed to import any qt binding
even though
This might also apply for other external matplotlib backends.
More details and example code will follow in the thread../pants run :bin
and it does not show the attached error.
Alternatively, running
./pants package :bin
dist/bin.pex
should work.
The zip file includes a minimal example of the scenario that I have in mind.
The dependency of matplotlib on PyQt5 is modeled manually in 3rdparty/BUILD
and running
./pants dependencies --transitive :bin
yields the expected result
//:example
3rdparty/requirements.txt:requirements.txt
3rdparty:requirements.txt#PyQt5
3rdparty:requirements.txt#matplotlib
However, during runtime, matplotlib does not seem to be able to find the backend in the part
matplotlib.use("Qt5Agg")
I've tried to set execution_mode = "venv"
on the target binary bin
, but that also doesn't help.
When I run this with a local venv instead:
python3.10 -m venv .venv3.10
source .venv3.10/bin/activate
pip install matplotlib PyQt5
python example.py
I get the expected result, that no exception is thrown and the plot is shown.enough-analyst-54434
07/22/2022, 7:54 AMpex --venv
venvs are created with sym-linked site-packages. They look like so:
$ tree -L 1 /home/jsirois/.pex/venvs/s/31b275de/venv/lib/python3.10/site-packages/
/home/jsirois/.pex/venvs/s/31b275de/venv/lib/python3.10/site-packages/
├── cycler-0.11.0.dist-info
├── cycler.py -> ../../../../../../installed_wheels/3a27e95f763a428a739d2add979fa7494c912a32c17c4c38c4d5f082cad165a3/cycler-0.11.0-py3-none-any.whl/cycler.py
├── dateutil -> ../../../../../../installed_wheels/961d03dc3453ebbc59dbdea9e4e11c5651520a876d0f4db161e8674aae935da9/python_dateutil-2.8.2-py2.py3-none-any.whl/dateutil
├── example.py
├── fontTools -> ../../../../../../installed_wheels/d73f25b283cd8033367451122aa868a23de0734757a01984e4b30b18b9050c72/fonttools-4.34.4-py3-none-any.whl/fontTools
├── fonttools-4.34.4.dist-info
├── kiwisolver -> ../../../../../../installed_wheels/55837ed1897eeab900d27dbbab9ea5ab6ff220e5fd31b2f6d93d093b06fb3e87/kiwisolver-1.4.4-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl/kiwisolver
├── kiwisolver-1.4.4.dist-info
├── matplotlib -> ../../../../../../installed_wheels/258ad4e7340db0bfe58794e6575a012db181204334d42bdf411fd4b248a0fd6f/matplotlib-3.5.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl/matplotlib
├── matplotlib-3.5.2.dist-info
├── matplotlib-3.5.2-py3.10-nspkg.pth -> ../../../../../../installed_wheels/258ad4e7340db0bfe58794e6575a012db181204334d42bdf411fd4b248a0fd6f/matplotlib-3.5.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl/matplotlib-3.5.2-py3.10-nspkg.pth
├── mpl_toolkits -> ../../../../../../installed_wheels/258ad4e7340db0bfe58794e6575a012db181204334d42bdf411fd4b248a0fd6f/matplotlib-3.5.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl/mpl_toolkits
├── numpy -> ../../../../../../installed_wheels/267e5995b4f8dcf6c4d713630d9f7a50a28520d8eea7dfb8953427b6258cfaea/numpy-1.23.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl/numpy
├── numpy-1.23.1.dist-info
├── numpy.libs -> ../../../../../../installed_wheels/267e5995b4f8dcf6c4d713630d9f7a50a28520d8eea7dfb8953427b6258cfaea/numpy-1.23.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl/numpy.libs
├── packaging -> ../../../../../../installed_wheels/ef103e05f519cdc783ae24ea4e2e0f508a9c99b2d4969652eed6a2e1ea5bd522/packaging-21.3-py3-none-any.whl/packaging
├── packaging-21.3.dist-info
├── pex-ns-pkgs
├── pex-ns-pkgs.pth
├── PIL -> ../../../../../../installed_wheels/22ba195f2a902be9d242fe8eff2373a1d698d401d926e49659fdff3bebd5c90c/Pillow-9.2.0-cp310-cp310-manylinux_2_28_x86_64.whl/PIL
├── Pillow-9.2.0.dist-info
├── Pillow.libs -> ../../../../../../installed_wheels/22ba195f2a902be9d242fe8eff2373a1d698d401d926e49659fdff3bebd5c90c/Pillow-9.2.0-cp310-cp310-manylinux_2_28_x86_64.whl/Pillow.libs
├── __pycache__
├── pylab.py -> ../../../../../../installed_wheels/258ad4e7340db0bfe58794e6575a012db181204334d42bdf411fd4b248a0fd6f/matplotlib-3.5.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl/pylab.py
├── pyparsing -> ../../../../../../installed_wheels/5026bae9a10eeaefb61dab2f09052b9f4307d44aee4eda64b309723d8d206bbc/pyparsing-3.0.9-py3-none-any.whl/pyparsing
├── pyparsing-3.0.9.dist-info
├── PyQt5 -> ../../../../../../installed_wheels/4170a485aa861e5662a082eadae57c1a4935570067e1213c09320a3cb8eae1b3/PyQt5-5.15.7-cp37-abi3-manylinux1_x86_64.whl/PyQt5
├── PyQt5-5.15.7.dist-info
├── python_dateutil-2.8.2.dist-info
├── six-1.16.0.dist-info
└── six.py -> ../../../../../../installed_wheels/8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254/six-1.16.0-py2.py3-none-any.whl/six.py
25 directories, 6 files
If you switch your pex_binary
back to execution_mode="venv"
(or use include_tools=True
) you can do this:
$ ./pants package :bin
01:46:05.19 [INFO] Wrote dist/bin.pex
$ PEX_TOOLS=1 ./dist/bin.pex venv dist/bin.venv
Now you have a venv that is not created out of symlinks:
$ tree -L 1 ./dist/bin.venv/lib/python3.10/site-packages/
./dist/bin.venv/lib/python3.10/site-packages/
├── cycler-0.11.0.dist-info
├── cycler.py
├── dateutil
├── example.py
├── fontTools
├── fonttools-4.34.4.dist-info
├── kiwisolver
├── kiwisolver-1.4.4.dist-info
├── matplotlib
├── matplotlib-3.5.2.dist-info
├── matplotlib-3.5.2-py3.10-nspkg.pth
├── mpl_toolkits
├── numpy
├── numpy-1.23.1.dist-info
├── numpy.libs
├── packaging
├── packaging-21.3.dist-info
├── PIL
├── Pillow-9.2.0.dist-info
├── Pillow.libs
├── pylab.py
├── pyparsing
├── pyparsing-3.0.9.dist-info
├── PyQt5
├── PyQt5-5.15.7.dist-info
├── PyQt5_Qt5-5.15.2.dist-info
├── PyQt5_sip-12.11.0.dist-info
├── python_dateutil-2.8.2.dist-info
├── six-1.16.0.dist-info
└── six.py
25 directories, 5 files
And matplotlib/pyqt5 seem to like that better:
$ ./dist/bin.venv/pex
Hello world
For a Pants `execution_mode="venv"`PEX, just pex --venv
is specified, we do not let you plumb the other two --venv-*
options:
$ pex -h
...
--venv [{prepend,append}]
Convert the pex file to a venv before executing it. If 'prepend' or 'append' is specified, then all scripts and console scripts provided by distributions in the pex file will be added to the PATH in the corresponding position. If the the pex file
will be run multiple times under a stable runtime PEX_ROOT, the venv creation will only be done once and subsequent runs will enjoy lower startup latency. (default: False)
--venv-copies, --no-venv-copies
If --venv is specified, create the venv using copies of base interpreter files instead of symlinks. This allows --venv mode PEXes to work across interpreter upgrades without being forced to remove the PEX_ROOT to allow the venv to re-build using
the upgraded interpreter. (default: False)
--venv-site-packages-copies, --no-venv-site-packages-copies
If --venv is specified, populate the venv site packages using hard links or copies of resolved PEX dependencies instead of symlinks. This can be used to work around problems with tools or libraries that are confused by symlinked source files.
(default: False)
...
What you'd need is the ability to set --venv-site-packages-copies
.
If using PEX_TOOLS=1 dist/bin.pex venv ...
isn't a good work-around for you @abundant-hospital-56388 please file a Pants issue to support pex --venv --venv-site-packages-copies
and we can get this in the 2.14.x series at the least.abundant-hospital-56388
08/17/2022, 12:45 PMenough-analyst-54434
08/17/2022, 3:38 PM"--include-tools"
and following the breadcrumbs.fresh-continent-76371
10/19/2022, 4:56 AMenough-analyst-54434
10/19/2022, 5:18 AMfresh-continent-76371
10/19/2022, 5:38 AM