https://pantsbuild.org/ logo
Powered by
#pex
Title
c

calm-artist-46894

05/04/2020, 4:59 PM
For packages like 
libcurl
that depends on shared library wich is not part of the Python package itself, what’s the recommended way to include them in a pex? Currently I try:
Copy code
pex --python=python3 pycurl -o testpycurl.pex
Then in a fresh Python3 container, I tried to use the pex:
Copy code
root@41161a1cada4:~# ./testpycurl.pex
Python 3.6.8 (default, Jun 11 2019, 01:21:42)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import pycurl
Traceback (most recent call last):
  File "<console>", line 1, in <module>
ImportError: libcurl.so.4: cannot open shared object file: No such file or directory
>>>
I understand this is beyond Python, i.e., 
pip install pycurl
itself would fail if 
libcurl
is not installed prior to install 
pycurl
, but would like to check if there any support from 
pex
, or this is believed to be solved by another layer of tools (e.g. docker)
a

aloof-angle-91616

05/04/2020, 5:02 PM
great question!!! pex files can contain bundled shared libraries, which should work across machines, although you might need to have some code to extract the shared library from the pex file first. that would be one way of working around a lack of libcurl on the target machine
docker is as you’ve mentioned another way to solve this
pants will often package necessary binaries like gcc so that they can be downloaded at a reliable location and version — if there’s a c or c++ library that is used often enough by pants users, we could consider making available osx and linux builds on the pantsbuild s3 and exposing that for pants users. but i don’t think that’d be in pex if we were to do something like that
w

witty-crayon-22786

05/04/2020, 5:14 PM
this question depends on the source of the wheels that end up in the pex
if the library in question is already a binary wheel on pypi, then it’s up to the folks who published it to do the right thing
if it’s a source tarball on pypi, then 
pex
will use 
pip
to build the binary wheel
in this case, it looks like 
pycurl
was pre-built: https://pypi.org/project/pycurl/#files
and when it was built, it was built to depend on shared libraries
it looks like it is possible to make the wheel more portable after the fact, and maybe 
pex
could do that…? https://stackoverflow.com/questions/23916186/how-to-include-external-library-with-python-wheel-package
a

aloof-angle-91616

05/04/2020, 5:18 PM
thank you so much for diving further into this question stu
that’s really, really interesting
pants currently doesn’t depend on anything besides the rust stdlib and rust packages or we might find it truly immediately useful
w

witty-crayon-22786

05/04/2020, 5:21 PM
pants does depend on some native wheels… we build them in travis.
a

aloof-angle-91616

05/04/2020, 5:21 PM
oh right ok
c

calm-artist-46894

05/04/2020, 5:41 PM
Thank @aloof-angle-91616 and @witty-crayon-22786. So, using 
pex
as it is now, I can 1. either build the wheel in a way that the needed 
.so
are embedded (
tensorflow
package is exactly this case) 2. or include the 
.so
file as loose file to the pex file, as well as some loading logic in my code
a

aloof-angle-91616

05/04/2020, 5:45 PM
i believe that is correct!
w

witty-crayon-22786

05/04/2020, 5:47 PM
re 2. : https://stackoverflow.com/questions/23916186/how-to-include-external-library-with-python-wheel-package is something else. it would basically be running a tool on the 
pycurl
wheel which would embed the shared library into the wheel, afaict. at that point pip/pex could use the new wheel, and it would be dependency-less
e

enough-analyst-54434

05/04/2020, 7:53 PM
Although there is an auditwheel for OSX (https://github.com/matthew-brett/delocate) having pex try to download and run these tools seems dicey. I just tried getting the pycurl example working using auditwheel and had no luck, but on seemingly unrelated mismatched openssl version errors between wheel build time and runtime. ... which is all to say, this sort of thing is dicey.
Powered by