Hello again I am having trouble using matplotlib for plottin

Question

Hello again I am having trouble using matplotlib for plotting with a Qt backend when run through pants or a generated pex file It looks like matplotlib is not finding other installed dependencies such as PyQt5 and that even when `execution mode = venv ` for the target In particular I get the error `ImportError Failed to import any qt binding` even though This might also apply for other external matplotlib backends More details and example code will follow in the thread

abundant-hospital-56388 · Answer

I am having the issue on Ubuntu 22 04 LTS Pants version 2 11 0 but also reproduced this for higher versions The expected behavior is that I can run ``` pants run bin``` and it does not show the attached error Alternatively running ``` pants package bin dist bin pex``` should work The zip file includes a minimal example of the scenario that I have in mind The dependency of matplotlib on PyQt5 is modeled manually in `3rdparty BUILD` and running ``` pants dependencies transitive bin``` yields the expected result ``` example 3rdparty requirements txt requirements txt 3rdparty requirements txt PyQt5 3rdparty requirements txt matplotlib``` However during runtime matplotlib does not seem to be able to find the backend in the part ```matplotlib use Qt5Agg ``` I ve tried to set `execution mode = venv ` on the target binary `bin` but that also doesn t help When I run this with a local venv instead ```python3 10 m venv venv3 10 source venv3 10 bin activate pip install matplotlib PyQt5 python example py``` I get the expected result that no exception is thrown and the plot is shown

enough-analyst-54434 · Answer

It appears matplotlib and or PyQt5 do not like the way `pex venv` venvs are created with sym linked site packages They look like so ```$ tree L 1 home jsirois pex venvs s 31b275de venv lib python3 10 site packages home jsirois pex venvs s 31b275de venv lib python3 10 site packages cycler 0 11 0 dist info cycler py gt installed wheels 3a27e95f763a428a739d2add979fa7494c912a32c17c4c38c4d5f082cad165a3 cycler 0 11 0 py3 none any whl cycler py dateutil gt installed wheels 961d03dc3453ebbc59dbdea9e4e11c5651520a876d0f4db161e8674aae935da9 python dateutil 2 8 2 py2 py3 none any whl dateutil example py fontTools gt installed wheels d73f25b283cd8033367451122aa868a23de0734757a01984e4b30b18b9050c72 fonttools 4 34 4 py3 none any whl fontTools fonttools 4 34 4 dist info kiwisolver gt installed wheels 55837ed1897eeab900d27dbbab9ea5ab6ff220e5fd31b2f6d93d093b06fb3e87 kiwisolver 1 4 4 cp310 cp310 manylinux 2 12 x86 64 manylinux2010 x86 64 whl kiwisolver kiwisolver 1 4 4 dist info matplotlib gt installed wheels 258ad4e7340db0bfe58794e6575a012db181204334d42bdf411fd4b248a0fd6f matplotlib 3 5 2 cp310 cp310 manylinux 2 17 x86 64 manylinux2014 x86 64 whl matplotlib matplotlib 3 5 2 dist info matplotlib 3 5 2 py3 10 nspkg pth gt installed wheels 258ad4e7340db0bfe58794e6575a012db181204334d42bdf411fd4b248a0fd6f matplotlib 3 5 2 cp310 cp310 manylinux 2 17 x86 64 manylinux2014 x86 64 whl matplotlib 3 5 2 py3 10 nspkg pth mpl toolkits gt installed wheels 258ad4e7340db0bfe58794e6575a012db181204334d42bdf411fd4b248a0fd6f matplotlib 3 5 2 cp310 cp310 manylinux 2 17 x86 64 manylinux2014 x86 64 whl mpl toolkits numpy gt installed wheels 267e5995b4f8dcf6c4d713630d9f7a50a28520d8eea7dfb8953427b6258cfaea numpy 1 23 1 cp310 cp310 manylinux 2 17 x86 64 manylinux2014 x86 64 whl numpy numpy 1 23 1 dist info numpy libs gt installed wheels 267e5995b4f8dcf6c4d713630d9f7a50a28520d8eea7dfb8953427b6258cfaea numpy 1 23 1 cp310 cp310 manylinux 2 17 x86 64 manylinux2014 x86 64 whl numpy libs packaging gt installed wheels ef103e05f519cdc783ae24ea4e2e0f508a9c99b2d4969652eed6a2e1ea5bd522 packaging 21 3 py3 none any whl packaging packaging 21 3 dist info pex ns pkgs pex ns pkgs pth PIL gt installed wheels 22ba195f2a902be9d242fe8eff2373a1d698d401d926e49659fdff3bebd5c90c Pillow 9 2 0 cp310 cp310 manylinux 2 28 x86 64 whl PIL Pillow 9 2 0 dist info Pillow libs gt installed wheels 22ba195f2a902be9d242fe8eff2373a1d698d401d926e49659fdff3bebd5c90c Pillow 9 2 0 cp310 cp310 manylinux 2 28 x86 64 whl Pillow libs pycache pylab py gt installed wheels 258ad4e7340db0bfe58794e6575a012db181204334d42bdf411fd4b248a0fd6f matplotlib 3 5 2 cp310 cp310 manylinux 2 17 x86 64 manylinux2014 x86 64 whl pylab py pyparsing gt installed wheels 5026bae9a10eeaefb61dab2f09052b9f4307d44aee4eda64b309723d8d206bbc pyparsing 3 0 9 py3 none any whl pyparsing pyparsing 3 0 9 dist info PyQt5 gt installed wheels 4170a485aa861e5662a082eadae57c1a4935570067e1213c09320a3cb8eae1b3 PyQt5 5 15 7 cp37 abi3 manylinux1 x86 64 whl PyQt5 PyQt5 5 15 7 dist info python dateutil 2 8 2 dist info six 1 16 0 dist info six py gt installed wheels 8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254 six 1 16 0 py2 py3 none any whl six py 25 directories 6 files``` If you switch your `pex binary` back to `execution mode= venv ` or use `include tools=True` you can do this ```$ pants package bin 01 46 05 19 INFO Wrote dist bin pex $ PEX TOOLS=1 dist bin pex venv dist bin venv``` Now you have a venv that is not created out of symlinks ```$ tree L 1 dist bin venv lib python3 10 site packages dist bin venv lib python3 10 site packages cycler 0 11 0 dist info cycler py dateutil example py fontTools fonttools 4 34 4 dist info kiwisolver kiwisolver 1 4 4 dist info matplotlib matplotlib 3 5 2 dist info matplotlib 3 5 2 py3 10 nspkg pth mpl toolkits numpy numpy 1 23 1 dist info numpy libs packaging packaging 21 3 dist info PIL Pillow 9 2 0 dist info Pillow libs pylab py pyparsing pyparsing 3 0 9 dist info PyQt5 PyQt5 5 15 7 dist info PyQt5 Qt5 5 15 2 dist info PyQt5 sip 12 11 0 dist info python dateutil 2 8 2 dist info six 1 16 0 dist info six py 25 directories 5 files``` And matplotlib pyqt5 seem to like that better ```$ dist bin venv pex Hello world``` For a Pants `execution mode= venv `PEX just `pex venv` is specified we do not let you plumb the other two ` venv ` options ```$ pex h venv prepend append Convert the pex file to a venv before executing it If prepend or append is specified then all scripts and console scripts provided by distributions in the pex file will be added to the PATH in the corresponding position If the the pex file will be run multiple times under a stable runtime PEX ROOT the venv creation will only be done once and subsequent runs will enjoy lower startup latency default False venv copies no venv copies If venv is specified create the venv using copies of base interpreter files instead of symlinks This allows venv mode PEXes to work across interpreter upgrades without being forced to remove the PEX ROOT to allow the venv to re build using the upgraded interpreter default False venv site packages copies no venv site packages copies If venv is specified populate the venv site packages using hard links or copies of resolved PEX dependencies instead of symlinks This can be used to work around problems with tools or libraries that are confused by symlinked source files default False ``` What you d need is the ability to set ` venv site packages copies` If using `PEX TOOLS=1 dist bin pex venv ` isn t a good work around for you < abundant hospital 56388> please file a Pants issue to support `pex venv venv site packages copies` and we can get this in the 2 14 x series at the least

enough-analyst-54434 · Answer

Actually I already filed that bug < abundant hospital 56388> please comment there if the work around is not sufficient and you need the feature

abundant-hospital-56388 · Answer

Hey < enough analyst 54434> I just responded to the github issue I don t think that the workaround described above is something that we can deal with long term and this seems to be the last thing that keeps us from fully switching over to pants for all our use cases

enough-analyst-54434 · Answer

Ok Eric pinged you on the GitHub issue thread about implementing the feature If you have interest its just a matter of grepping for another similar Pex flag say ` include tools ` and following the breadcrumbs

fresh-continent-76371 · Answer

< enough analyst 54434> would you have some suggestions as to how to solve the same for tests that are running so many of the tests I am dealing with are depending on pyqt5 and the` pants test` PEX built for testing i think are suffering this same issue I can see in the git issue for the pex binary I the execution mode but I do not see the same on a test execution or know where it goes

enough-analyst-54434 · Answer

For tests and all internal PEXes that would be <https www pantsbuild org docs reference pex venv use symlinks|https www pantsbuild org docs reference pex venv use symlinks> But that defaults the way you want no symlinks so you re likely hitting a different issue The best bet is filing a well documented issue with full command lines and output where possible

fresh-continent-76371 · Answer

Thanks < enough analyst 54434> I will keep digging