For packages like `libcurl` that depends on shared...
# pex
For packages like
that depends on shared library wich is not part of the Python package itself, what’s the recommended way to include them in a pex? Currently I try:
Copy code
pex --python=python3 pycurl -o testpycurl.pex
Then in a fresh Python3 container, I tried to use the pex:
Copy code
root@41161a1cada4:~# ./testpycurl.pex
Python 3.6.8 (default, Jun 11 2019, 01:21:42)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pycurl
Traceback (most recent call last):
  File "<console>", line 1, in <module>
ImportError: cannot open shared object file: No such file or directory
I understand this is beyond Python, i.e.,
pip install pycurl
itself would fail if
is not installed prior to install
, but would like to check if there any support from
, or this is believed to be solved by another layer of tools (e.g. docker)
great question!!! pex files can contain bundled shared libraries, which should work across machines, although you might need to have some code to extract the shared library from the pex file first. that would be one way of working around a lack of libcurl on the target machine
docker is as you’ve mentioned another way to solve this
pants will often package necessary binaries like gcc so that they can be downloaded at a reliable location and version — if there’s a c or c++ library that is used often enough by pants users, we could consider making available osx and linux builds on the pantsbuild s3 and exposing that for pants users. but i don’t think that’d be in pex if we were to do something like that
this question depends on the source of the wheels that end up in the pex
if the library in question is already a binary wheel on pypi, then it’s up to the folks who published it to do the right thing
if it’s a source tarball on pypi, then
will use
to build the binary wheel
in this case, it looks like
was pre-built:
and when it was built, it was built to depend on shared libraries
it looks like it is possible to make the wheel more portable after the fact, and maybe
could do that…?
thank you so much for diving further into this question stu
that’s really, really interesting
pants currently doesn’t depend on anything besides the rust stdlib and rust packages or we might find it truly immediately useful
pants does depend on some native wheels… we build them in travis.
oh right ok
Thank @aloof-angle-91616 and @witty-crayon-22786. So, using
as it is now, I can 1. either build the wheel in a way that the needed
are embedded (
package is exactly this case) 2. or include the
file as loose file to the pex file, as well as some loading logic in my code
i believe that is correct!
re 2. : is something else. it would basically be running a tool on the
wheel which would embed the shared library into the wheel, afaict. at that point pip/pex could use the new wheel, and it would be dependency-less
Although there is an auditwheel for OSX ( having pex try to download and run these tools seems dicey. I just tried getting the pycurl example working using auditwheel and had no luck, but on seemingly unrelated mismatched openssl version errors between wheel build time and runtime. ... which is all to say, this sort of thing is dicey.