Hello again I am having trouble using matplotlib for plottin Pants #general

Hello again, I am having trouble using matplotlib ...

abundant-hospital-56388

07/21/2022, 8:31 AM

Hello again, I am having trouble using matplotlib for plotting with a Qt backend when run through pants or a generated .pex file. It looks like matplotlib is not finding other installed dependencies such as PyQt5 and that even when

execution_mode = "venv"

for the target. In particular I get the error

ImportError: Failed to import any qt binding

even though This might also apply for other external matplotlib backends. More details and example code will follow in the thread.

abundant-hospital-56388

07/21/2022, 8:40 AM

I am having the issue on Ubuntu 22.04 LTS. Pants version 2.11.0, but also reproduced this for higher versions. The expected behavior is that I can run

Copy code

./pants run :bin

and it does not show the attached error. Alternatively, running

Copy code

./pants package :bin
dist/bin.pex

should work. The zip file includes a minimal example of the scenario that I have in mind. The dependency of matplotlib on PyQt5 is modeled manually in

3rdparty/BUILD

and running

Copy code

./pants dependencies --transitive :bin

yields the expected result

Copy code

//:example
3rdparty/requirements.txt:requirements.txt
3rdparty:requirements.txt#PyQt5
3rdparty:requirements.txt#matplotlib

However, during runtime, matplotlib does not seem to be able to find the backend in the part

Copy code

matplotlib.use("Qt5Agg")

I've tried to set

execution_mode = "venv"

on the target binary

bin

, but that also doesn't help. When I run this with a local venv instead:

Copy code

python3.10 -m venv .venv3.10
source .venv3.10/bin/activate
pip install matplotlib PyQt5
python example.py

I get the expected result, that no exception is thrown and the plot is shown.

matplotlib-qt.zip error.txt

enough-analyst-54434

07/22/2022, 7:54 AM

It appears matplotlib and/or PyQt5 do not like the way

pex --venv

venvs are created with sym-linked site-packages. They look like so:

Copy code

$ tree -L 1 /home/jsirois/.pex/venvs/s/31b275de/venv/lib/python3.10/site-packages/
/home/jsirois/.pex/venvs/s/31b275de/venv/lib/python3.10/site-packages/
├── cycler-0.11.0.dist-info
├── cycler.py -> ../../../../../../installed_wheels/3a27e95f763a428a739d2add979fa7494c912a32c17c4c38c4d5f082cad165a3/cycler-0.11.0-py3-none-any.whl/cycler.py
├── dateutil -> ../../../../../../installed_wheels/961d03dc3453ebbc59dbdea9e4e11c5651520a876d0f4db161e8674aae935da9/python_dateutil-2.8.2-py2.py3-none-any.whl/dateutil
├── example.py
├── fontTools -> ../../../../../../installed_wheels/d73f25b283cd8033367451122aa868a23de0734757a01984e4b30b18b9050c72/fonttools-4.34.4-py3-none-any.whl/fontTools
├── fonttools-4.34.4.dist-info
├── kiwisolver -> ../../../../../../installed_wheels/55837ed1897eeab900d27dbbab9ea5ab6ff220e5fd31b2f6d93d093b06fb3e87/kiwisolver-1.4.4-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl/kiwisolver
├── kiwisolver-1.4.4.dist-info
├── matplotlib -> ../../../../../../installed_wheels/258ad4e7340db0bfe58794e6575a012db181204334d42bdf411fd4b248a0fd6f/matplotlib-3.5.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl/matplotlib
├── matplotlib-3.5.2.dist-info
├── matplotlib-3.5.2-py3.10-nspkg.pth -> ../../../../../../installed_wheels/258ad4e7340db0bfe58794e6575a012db181204334d42bdf411fd4b248a0fd6f/matplotlib-3.5.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl/matplotlib-3.5.2-py3.10-nspkg.pth
├── mpl_toolkits -> ../../../../../../installed_wheels/258ad4e7340db0bfe58794e6575a012db181204334d42bdf411fd4b248a0fd6f/matplotlib-3.5.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl/mpl_toolkits
├── numpy -> ../../../../../../installed_wheels/267e5995b4f8dcf6c4d713630d9f7a50a28520d8eea7dfb8953427b6258cfaea/numpy-1.23.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl/numpy
├── numpy-1.23.1.dist-info
├── numpy.libs -> ../../../../../../installed_wheels/267e5995b4f8dcf6c4d713630d9f7a50a28520d8eea7dfb8953427b6258cfaea/numpy-1.23.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl/numpy.libs
├── packaging -> ../../../../../../installed_wheels/ef103e05f519cdc783ae24ea4e2e0f508a9c99b2d4969652eed6a2e1ea5bd522/packaging-21.3-py3-none-any.whl/packaging
├── packaging-21.3.dist-info
├── pex-ns-pkgs
├── pex-ns-pkgs.pth
├── PIL -> ../../../../../../installed_wheels/22ba195f2a902be9d242fe8eff2373a1d698d401d926e49659fdff3bebd5c90c/Pillow-9.2.0-cp310-cp310-manylinux_2_28_x86_64.whl/PIL
├── Pillow-9.2.0.dist-info
├── Pillow.libs -> ../../../../../../installed_wheels/22ba195f2a902be9d242fe8eff2373a1d698d401d926e49659fdff3bebd5c90c/Pillow-9.2.0-cp310-cp310-manylinux_2_28_x86_64.whl/Pillow.libs
├── __pycache__
├── pylab.py -> ../../../../../../installed_wheels/258ad4e7340db0bfe58794e6575a012db181204334d42bdf411fd4b248a0fd6f/matplotlib-3.5.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl/pylab.py
├── pyparsing -> ../../../../../../installed_wheels/5026bae9a10eeaefb61dab2f09052b9f4307d44aee4eda64b309723d8d206bbc/pyparsing-3.0.9-py3-none-any.whl/pyparsing
├── pyparsing-3.0.9.dist-info
├── PyQt5 -> ../../../../../../installed_wheels/4170a485aa861e5662a082eadae57c1a4935570067e1213c09320a3cb8eae1b3/PyQt5-5.15.7-cp37-abi3-manylinux1_x86_64.whl/PyQt5
├── PyQt5-5.15.7.dist-info
├── python_dateutil-2.8.2.dist-info
├── six-1.16.0.dist-info
└── six.py -> ../../../../../../installed_wheels/8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254/six-1.16.0-py2.py3-none-any.whl/six.py

25 directories, 6 files

If you switch your

pex_binary

back to

execution_mode="venv"

(or use

include_tools=True

) you can do this:

Copy code

$ ./pants package :bin
01:46:05.19 [INFO] Wrote dist/bin.pex
$ PEX_TOOLS=1 ./dist/bin.pex venv dist/bin.venv

Now you have a venv that is not created out of symlinks:

Copy code

$ tree -L 1 ./dist/bin.venv/lib/python3.10/site-packages/
./dist/bin.venv/lib/python3.10/site-packages/
├── cycler-0.11.0.dist-info
├── cycler.py
├── dateutil
├── example.py
├── fontTools
├── fonttools-4.34.4.dist-info
├── kiwisolver
├── kiwisolver-1.4.4.dist-info
├── matplotlib
├── matplotlib-3.5.2.dist-info
├── matplotlib-3.5.2-py3.10-nspkg.pth
├── mpl_toolkits
├── numpy
├── numpy-1.23.1.dist-info
├── numpy.libs
├── packaging
├── packaging-21.3.dist-info
├── PIL
├── Pillow-9.2.0.dist-info
├── Pillow.libs
├── pylab.py
├── pyparsing
├── pyparsing-3.0.9.dist-info
├── PyQt5
├── PyQt5-5.15.7.dist-info
├── PyQt5_Qt5-5.15.2.dist-info
├── PyQt5_sip-12.11.0.dist-info
├── python_dateutil-2.8.2.dist-info
├── six-1.16.0.dist-info
└── six.py

25 directories, 5 files

And matplotlib/pyqt5 seem to like that better:

Copy code

$ ./dist/bin.venv/pex
Hello world

For a Pants `execution_mode="venv"`PEX, just

pex --venv

is specified, we do not let you plumb the other two

--venv-*

options:

Copy code

$ pex -h
...
  --venv [{prepend,append}]
                        Convert the pex file to a venv before executing it. If 'prepend' or 'append' is specified, then all scripts and console scripts provided by distributions in the pex file will be added to the PATH in the corresponding position. If the the pex file
                        will be run multiple times under a stable runtime PEX_ROOT, the venv creation will only be done once and subsequent runs will enjoy lower startup latency. (default: False)
  --venv-copies, --no-venv-copies
                        If --venv is specified, create the venv using copies of base interpreter files instead of symlinks. This allows --venv mode PEXes to work across interpreter upgrades without being forced to remove the PEX_ROOT to allow the venv to re-build using
                        the upgraded interpreter. (default: False)
  --venv-site-packages-copies, --no-venv-site-packages-copies
                        If --venv is specified, populate the venv site packages using hard links or copies of resolved PEX dependencies instead of symlinks. This can be used to work around problems with tools or libraries that are confused by symlinked source files.
                        (default: False)
...

What you'd need is the ability to set

--venv-site-packages-copies

. If using

PEX_TOOLS=1 dist/bin.pex venv ...

isn't a good work-around for you @abundant-hospital-56388 please file a Pants issue to support

pex --venv --venv-site-packages-copies

and we can get this in the 2.14.x series at the least.

enough-analyst-54434

07/22/2022, 2:33 PM

Actually, I already filed that bug: https://github.com/pantsbuild/pants/issues/14559 @abundant-hospital-56388 please comment there if the work around is not sufficient and you need the feature.

abundant-hospital-56388

08/17/2022, 12:45 PM

Hey @enough-analyst-54434, I just responded to the github issue. I don't think that the workaround described above is something that we can deal with long-term and this seems to be the last thing that keeps us from fully switching over to pants for all our use cases

enough-analyst-54434

08/17/2022, 3:38 PM

Ok. Eric pinged you on the GitHub issue thread about implementing the feature. If you have interest its just a matter of grepping for another similar Pex flag, say

"--include-tools"

and following the breadcrumbs.

fresh-continent-76371

10/19/2022, 4:56 AM

@enough-analyst-54434 - would you have some suggestions, as to how to solve the same for "tests" that are running .. so many of the tests I am dealing with are depending on pyqt5 and the`./pants test` PEX' built for testing, i think are suffering this same issue) I can see in the git issue, for the pex binary I the execution_mode.. but I do not see the same on a test execution (or know where it goes)

enough-analyst-54434

10/19/2022, 5:18 AM

For tests and all internal PEXes, that would be https://www.pantsbuild.org/docs/reference-pex#venv_use_symlinks But that defaults the way you want (no symlinks); so you're likely hitting a different issue. The best bet is filing a well documented issue with full command lines and output where possible.

👍 1

fresh-continent-76371

10/19/2022, 5:38 AM

Thanks @enough-analyst-54434 - I will keep digging

4 Views

Open in Slack

Previous Next