https://pantsbuild.org/ logo
#general
Title
# general
e

eager-dress-66405

11/05/2021, 7:04 AM
Any one have any hints on what be needed to get pants to work with
pybedtools
?
Copy code
mpcusack@mpcp15:~/dev/color$ ./pants version
2.7.1
Copy code
mpcusack@mpcp15:~/dev/color$ grep "interpreter_constraints " pants.toml 
interpreter_constraints = ["CPython==3.6.*"]
Copy code
mpcusack@mpcp15:~/dev/color$ grep pybedtool constraints.lock 
pybedtools==0.8.2
Copy code
mpcusack@mpcp15:~/dev/color$ cat src/bin/tests/test_pybedtools.py 
import pybedtools


def test_pybedtools():
    ...
Copy code
mpcusack@mpcp15:~/dev/color$ ./pants test src/bin/tests/test_pybedtools.py 
00:00:53.66 [ERROR] Completed: test - src/bin/tests/test_pybedtools.py failed (exit code 2).
============================= test session starts ==============================
platform linux -- Python 3.6.15, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
rootdir: /tmp/process-executionoQQWar, configfile: pyproject.toml
plugins: cov-3.0.0
collected 0 items / 1 error

==================================== ERRORS ====================================
______________ ERROR collecting src/bin/tests/test_pybedtools.py _______________
ImportError while importing test module '/tmp/process-executionoQQWar/src/bin/tests/test_pybedtools.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/home/mpcusack/.pyenv/versions/3.6.15/lib/python3.6/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
src/bin/tests/test_pybedtools.py:1: in <module>
    import pybedtools
/home/mpcusack/.cache/pants/named_caches/pex_root/venvs/short/ed00b369/lib/python3.6/site-packages/pybedtools/__init__.py:9: in <module>
    from .cbedtools import (
E   ImportError: /home/mpcusack/.cache/pants/named_caches/pex_root/venvs/short/ed00b369/lib/python3.6/site-packages/pybedtools/cbedtools.cpython-36m-x86_64-linux-gnu.so: undefined symbol: PyFPE_jbuf
---- generated xml file: /tmp/process-executionoQQWar/var/pytest-report.xml ----
=========================== short test summary info ============================
ERROR src/bin/tests/test_pybedtools.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.09s ===============================



š„‚ src/bin/tests/test_pybedtools.py failed.
Copy code
(virtualenv3) mpcusack@mpcp15:~/dev/color$ python
Python 3.6.15 (default, Sep 10 2021, 00:26:58) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pybedtools
>>>
h

happy-kitchen-89482

11/05/2021, 12:37 PM
Hmm, Google tells me that
undefined symbol PyFPE_jbuf
is a fairly ubiquitous problem in several contexts, unrelated to Pants. So there is some difference in environments between the Pants case and the raw case.
Is this the exact same interpreter? (e.g., can you try
/home/mpcusack/.pyenv/versions/3.6.15/python -c "import pybedtools"
)
And, can you try
import fpectl
in both cases? (Pants test and raw interpreter)?
Discussions like this one lead me to believe that this is due to building the pybedtools sdist on an interpreter with fpectl and then running it on one without fpectl.
e

eager-dress-66405

11/05/2021, 8:12 PM
Hi Benjy. Thanks for looking at this. I think it's something to do with the way pants/pex is building/installing the module, but I'm not sure where.
Copy code
mpcusack@mpcp15:~$ pyenv activate pybedtoolstest
(pybedtoolstest) mpcusack@mpcp15:~$ python -V
Python 3.6.15
(pybedtoolstest) mpcusack@mpcp15:~$ pip install pybedtools==0.8.2
Collecting pybedtools
  Using cached <https://files.pythonhosted.org/packages/5b/57/46e9c8135954550d1d3a05db5b0462a264239f7f71c78af304bb47c5c900/pybedtools-0.8.2.tar.gz>
Collecting six (from pybedtools)
  Using cached <https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl>
Collecting pysam (from pybedtools)
  Using cached <https://files.pythonhosted.org/packages/03/ef/6e7342dbe0571fa078bd312f07dcb9b0693b89f8f6fbdf7876e4423374b9/pysam-0.17.0-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.whl>
Installing collected packages: six, pysam, pybedtools
  Running setup.py install for pybedtools ... done
Successfully installed pybedtools-0.8.2 pysam-0.17.0 six-1.16.0
(pybedtoolstest) mpcusack@mpcp15:~$ python -c "import pybedtools"
(pybedtoolstest) mpcusack@mpcp15:~$ echo $?
0
h

happy-kitchen-89482

11/06/2021, 12:12 AM
Can you try the
import fpectl
thing in each environment?
e

eager-dress-66405

11/06/2021, 2:51 AM
For fpectl: Trying to import from the pyenv python 3.6.15 virtual env.
Copy code
(pybedtoolstest) mpcusack@mpcp15:~$ python -c "import fpectl"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'fpectl'
From pants:
Copy code
mpcusack@mpcp15:~/dev/color$ ./pants test   src/bin/tests/test_fpectl.py 
19:50:22.53 [WARN] No `.python-version` file found in the build root, but <PYENV_LOCAL> was set in `[python-setup].interpreter_search_paths`.
19:50:24.65 [INFO] Completed: Building requirements.pex
19:50:27.70 [INFO] Completed: Building pytest.pex from pytest_default_lockfile.txt
19:50:28.37 [INFO] Completed: Building pytest_runner.pex
19:50:28.75 [ERROR] Completed: test - src/bin/tests/test_fpectl.py failed (exit code 2).
============================= test session starts ==============================
platform linux -- Python 3.6.15, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
rootdir: /tmp/process-executionxWqBky, configfile: pyproject.toml
plugins: cov-3.0.0
collected 0 items / 1 error

==================================== ERRORS ====================================
________________ ERROR collecting src/bin/tests/test_fpectl.py _________________
ImportError while importing test module '/tmp/process-executionxWqBky/src/bin/tests/test_fpectl.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/home/mpcusack/.pyenv/versions/3.6.15/lib/python3.6/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
src/bin/tests/test_fpectl.py:1: in <module>
    import fpectl
E   ModuleNotFoundError: No module named 'fpectl'
---- generated xml file: /tmp/process-executionxWqBky/var/pytest-report.xml ----
=========================== short test summary info ============================
ERROR src/bin/tests/test_fpectl.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.08s ===============================



š„‚ src/bin/tests/test_fpectl.py failed.
mpcusack@mpcp15:~/dev/color$ cat src/bin/tests/test_fpectl.py
import fpectl


def test_fpectl():
    ...
I'm actually unable to get the
fpectl
module to be accessible no matter what I do
Copy code
mpcusack@mpcp15:~/dev/color$ PYTHON_CONFIGURE_OPTS="--with-fpectl" pyenv install 3.6.15
pyenv: /home/mpcusack/.pyenv/versions/3.6.15 already exists
continue with installation? (y/N) y
Installing Python-3.6.15...
Installed Python-3.6.15 to /home/mpcusack/.pyenv/versions/3.6.15

mpcusack@mpcp15:~/dev/color$ python -c "import fpectl"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'fpectl'
However now that I have built python 3.6.15 with
--with-fpectl
I can use
pybedtools
from pants
Copy code
āœ“ src/bin/tests/test_pybedtools.py succeeded.
š„‚ src/bin/tests/test_fpectl.py failed.
The original 3.6.15, without
--with-fpectl
, virtualenv with
pybedtools
pip installed worked though, could pants be selecting the wrong wheel to install?
h

happy-kitchen-89482

11/06/2021, 9:24 AM
AFAICT
pybedtools
is distributed only as an sdist, so the problem is related to building wheels rather than selecting existing ones (and in any case I don't think there is an environment marker to denote the presence/absence of fpectl?)
It's good that things are working now, but this is still a bit of a heisenbug. I don't think this is related specifically to Pants, but more to the fact that you had on your system one interpreter with fpectl and one without, and happened to build and cache a wheel on one and then use it on the other.
E.g., possibly system interpreter vs pyenv interpreter
So I think to make the solution robust it might make sense to pick one destiny (with fpectl or without, and I think without is considered safer), ensure that all interpreters pants can discover conform to that, then nuke
~/.cache/pants/named_caches/pex_root/venvs
and rebuild all the wheels.
E.g., one way to ensure that all interpreters pants can discover are conformant is to set
interpreter_search_path
to
"['PYENV']"
, so it will only consider pyenv interpreters. I don't know if that is a change you can make globally though.
Were you encountering this problem just on one machine, or was this widespread? If it was just one machine, and you have a fix, maybe that's sufficient.
Anyway, that is my best understanding of the problem. Arguably you may want to get this working without fpectl, or maybe you're ok as-is šŸ™‚
e

eager-dress-66405

11/08/2021, 1:33 PM
So far just from my machine, but I will see of others can replicate it. FWIW our interpreter_search_path does not include PATH. It only includes ASDF and PYENV, and I only use
pyenv
. I only have 2 pythons installed via pyenv (3.6.15 & 3.9.7), both of which were (originally) installed without any special env vars/flags.
So make things extra confusing I no longer can replicate the failure. I reinstalled 3.6.15 w/o the
--with-fpectl
and it continues to work....
h

happy-kitchen-89482

11/08/2021, 1:34 PM
Hmmm, and Pants uses its own pip cache, so you shouldn't be contaminated by outside pip runs
Very curious
5 Views