Hi all! I use pants package to get a .pex python binary. In requirements file I have a dependency: gmsh==4.12.1 When I execute the binary, I see the following problem

Warning: could not find Gmsh shared library libgmsh.so.4.12 with ctypes.util.find_library() or in the following locations: ['/root/.pex/installed_wheels/5b4820c98b3a9fff6fe08010f8dfd52a77ca6ecc46b0a1212245061298862c60/gmsh-4.12.1-py2.py3-none-manylinux_2_24_x86_64.whl/libgmsh.so.4.12', '/root/.pex/installed_wheels/5b4820c98b3a9fff6fe08010f8dfd52a77ca6ecc46b0a1212245061298862c60/gmsh-4.12.1-py2.py3-none-manylinux_2_24_x86_64.whl/lib/libgmsh.so.4.12', '/root/.pex/installed_wheels/5b4820c98b3a9fff6fe08010f8dfd52a77ca6ecc46b0a1212245061298862c60/gmsh-4.12.1-py2.py3-none-manylinux_2_24_x86_64.whl/Lib/libgmsh.so.4.12', '/root/.pex/installed_wheels/5b4820c98b3a9fff6fe08010f8dfd52a77ca6ecc46b0a1212245061298862c60/libgmsh.so.4.12', '/root/.pex/installed_wheels/5b4820c98b3a9fff6fe08010f8dfd52a77ca6ecc46b0a1212245061298862c60/lib/libgmsh.so.4.12', '/root/.pex/installed_wheels/5b4820c98b3a9fff6fe08010f8dfd52a77ca6ecc46b0a1212245061298862c60/Lib/libgmsh.so.4.12', '/root/.pex/installed_wheels/libgmsh.so.4.12', '/root/.pex/installed_wheels/lib/libgmsh.so.4.12', '/root/.pex/installed_wheels/Lib/libgmsh.so.4.12']

When I expect the mentioned directory

'/root/.pex/installed_wheels/5b4820c98b3a9fff6fe08010f8dfd52a77ca6ecc46b0a1212245061298862c60/gmsh-4.12.1-py2.py3-none-manylinux_2_24_x86_64.whl

I can see indeed that the library is not in the root directory or /lib subdir. It is in a

/.prefix/lib

subdir. In the root dir there is a

.layout.json

file located. Contents are

{"record_relpath": "gmsh-4.12.1.dist-info/RECORD", "stash_dir": ".prefix"}

I am not sure how to debug it further, or how to instruct the library look at the .layout file, or how to instruct pants to have save executable in the root directory. Any pointers would be appreciated!

Thanks for the thorough diagnosis! I can reproduce this just in Pex:

$ pex gmsh==4.12.1 -o out.pex
$ ./out.pex
Python 3.11.2 (main, Feb 23 2023, 00:15:28) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import gmsh
Warning: could not find Gmsh shared library libgmsh.4.12.dylib with ctypes.util.find_library() or in the following locations: ['/Users/benjyw/.pex/installed_wheels/eaf48bbed0f09c6f731e47ed0dbd184fef0dd8cf3b9b534222229a2b910d4c6a/gmsh-4.12.1-py2.py3-none-macosx_12_0_arm64.whl/libgmsh.4.12.dylib', '/Users/benjyw/.pex/installed_wheels/eaf48bbed0f09c6f731e47ed0dbd184fef0dd8cf3b9b534222229a2b910d4c6a/gmsh-4.12.1-py2.py3-none-macosx_12_0_arm64.whl/lib/libgmsh.4.12.dylib', '/Users/benjyw/.pex/installed_wheels/eaf48bbed0f09c6f731e47ed0dbd184fef0dd8cf3b9b534222229a2b910d4c6a/gmsh-4.12.1-py2.py3-none-macosx_12_0_arm64.whl/Lib/libgmsh.4.12.dylib', '/Users/benjyw/.pex/installed_wheels/eaf48bbed0f09c6f731e47ed0dbd184fef0dd8cf3b9b534222229a2b910d4c6a/libgmsh.4.12.dylib', '/Users/benjyw/.pex/installed_wheels/eaf48bbed0f09c6f731e47ed0dbd184fef0dd8cf3b9b534222229a2b910d4c6a/lib/libgmsh.4.12.dylib', '/Users/benjyw/.pex/installed_wheels/eaf48bbed0f09c6f731e47ed0dbd184fef0dd8cf3b9b534222229a2b910d4c6a/Lib/libgmsh.4.12.dylib', '/Users/benjyw/.pex/installed_wheels/libgmsh.4.12.dylib', '/Users/benjyw/.pex/installed_wheels/lib/libgmsh.4.12.dylib', '/Users/benjyw/.pex/installed_wheels/Lib/libgmsh.4.12.dylib']
>>>

Mine is Ubuntu. Yes, exactly that seems to be the case (the library is in .prefix) :(

I note further that the Python wrapper for gmsh seems to have problems even in a vanilla Python virtualenv: Nope, that was me fat-fingering something, I tried from scratch and it does seem to work in a virtualenv.

I was clicking through other dependencies in installed_wheels and noticed that a few other also have the location file with .prefix. Also, when I run the app as pants run my_app.py - not phe .pex file , there is no problem (it seems to me)

Hmm, in a virtualenv I notice that the dylib is in an expected location:

/tmp/venv/lib/libgmsh.4.12.dylib

2.1.137

Looking at the pex sources, I think I have a vague clue, but I'll need to dive into the internals a bit.

Issue opened: https://github.com/pantsbuild/pex/issues/2336

Thanks!

Thank you for looking into it!

@adamant-wolf-51914 See my update on the issue, with a suggested workaround