https://pantsbuild.org/ logo
Join Slack
Powered by
There is another Python package import issue I cam...
# general
r
There is another Python package import issue I came across: Is is possible to use system installed site-packages (and make them available inside pants)? More details in the 🧵.
The product uses the BPF Compiler Collection (BCC) package in order to use the Berkley Package Filter (BPF). This package cannot be installed by pip, as it is deeply bound to the kernel and needs to be compiled as kernel module. Thus it need to be system installed. On system python it is usable with 
from bcc import BPF
. Note, that this is not to be confused with the physics module regarding the BCC lattice, which must not be installed simultaniously. With venv there is an option 
--system-site-packages
to allow such packages to be used. How would I be able to use this system installed package with Pants?
I am not sure whether local requirements (from docu) supports unpacked site-packages. The following requirement does not seem to work for me.
Copy code
python_requirement(
    name="bcc",
    requirements=[
        "bcc @ file:///usr/lib/python3.10/site-packages/bcc/",
    ],
)
e
In short, you can't. The reason is this breaks a key principle in Pants - repeatable builds. Passes on your machine with that system site library installed, not on other machines without that.
So ... if its installed in system site packages it must be generally installable in any site packages?
Ok, it is, but you have to do some extra work. I followed the instructions here (I'm on Arch): https://github.com/iovisor/bcc/blob/master/INSTALL.md#arch---source That left me with an sdist:
Copy code
$ ls -l bcc/build/src/python/bcc-python3/dist/
total 44
-rw-r--r-- 1 jsirois jsirois 42837 Jan 11 07:47 bcc-0.23.0-23a21423.tar.gz
That sdist can be then placed on an internal repo and you can configure Pants to use that repo (or just use a direct reference url which sounds like its doing the trick for your investigation.
This will allow building a PEX uniformly via Pants. It's true that the Pex will only successfully run on machines with the rest of bcc installed (the native libs bits).
Yeah, the sdist is pure python. It uses ctypes to load the shared library that must be pre-installed on the machine:
Copy code
$ head -23 bcc/src/python/bcc/libbcc.py  | tail -9
import ctypes as ct

lib = ct.CDLL("libbcc.so.0", use_errno=True)

# needed for perf_event_attr() ctype
from .perf import Perf

# keep in sync with bcc_common.h
lib.bpf_module_create_c.restype = ct.c_void_p
r
Thanks for your quick response! I believe I get the same sdist files when installing without the manual compilation.
Copy code
/usr/lib/python3.10/site-packages/bcc$ tree
.
├── containers.py
├── disassembler.py
├── __init__.py
├── libbcc.py
├── perf.py
├── __pycache__
│   ├── containers.cpython-310.pyc
│   ├── disassembler.cpython-310.pyc
│   ├── __init__.cpython-310.pyc
│   ├── libbcc.cpython-310.pyc
│   ├── perf.cpython-310.pyc
│   ├── syscall.cpython-310.pyc
│   ├── table.cpython-310.pyc
│   ├── tcp.cpython-310.pyc
│   ├── usdt.cpython-310.pyc
│   ├── utils.cpython-310.pyc
│   └── version.cpython-310.pyc
├── syscall.py
├── table.py
├── tcp.py
├── usdt.py
├── utils.py
└── version.py

1 directory, 22 files
e
You do, but the problem is you need a distribution, not an installed distribution. Your attempt was valiant but didn't work because a direct url reference needs to point at a file and not a directory.
r
Ah, I see, so the manual compilation will also give a distributable, which contains the appropriate files and can than itself be linked.
Thanks again a lot for your help! 🙏 I will try the self compilation next.
Thank you so much and have a pleasant day!
➕ 2
3 Views
Open in Slack
PreviousNext
Powered by