https://pantsbuild.org/ logo
Join Slack
Powered by
Any one have any hints on what be needed to get pa...
# general
e
Any one have any hints on what be needed to get pants to work with 
pybedtools
?
Copy code
mpcusack@mpcp15:~/dev/color$ ./pants version
2.7.1
Copy code
mpcusack@mpcp15:~/dev/color$ grep "interpreter_constraints " pants.toml 
interpreter_constraints = ["CPython==3.6.*"]
Copy code
mpcusack@mpcp15:~/dev/color$ grep pybedtool constraints.lock 
pybedtools==0.8.2
Copy code
mpcusack@mpcp15:~/dev/color$ cat src/bin/tests/test_pybedtools.py 
import pybedtools


def test_pybedtools():
    ...
Copy code
mpcusack@mpcp15:~/dev/color$ ./pants test src/bin/tests/test_pybedtools.py 
00:00:53.66 [ERROR] Completed: test - src/bin/tests/test_pybedtools.py failed (exit code 2).
============================= test session starts ==============================
platform linux -- Python 3.6.15, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
rootdir: /tmp/process-executionoQQWar, configfile: pyproject.toml
plugins: cov-3.0.0
collected 0 items / 1 error

==================================== ERRORS ====================================
______________ ERROR collecting src/bin/tests/test_pybedtools.py _______________
ImportError while importing test module '/tmp/process-executionoQQWar/src/bin/tests/test_pybedtools.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/home/mpcusack/.pyenv/versions/3.6.15/lib/python3.6/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
src/bin/tests/test_pybedtools.py:1: in <module>
    import pybedtools
/home/mpcusack/.cache/pants/named_caches/pex_root/venvs/short/ed00b369/lib/python3.6/site-packages/pybedtools/__init__.py:9: in <module>
    from .cbedtools import (
E   ImportError: /home/mpcusack/.cache/pants/named_caches/pex_root/venvs/short/ed00b369/lib/python3.6/site-packages/pybedtools/cbedtools.cpython-36m-x86_64-linux-gnu.so: undefined symbol: PyFPE_jbuf
---- generated xml file: /tmp/process-executionoQQWar/var/pytest-report.xml ----
=========================== short test summary info ============================
ERROR src/bin/tests/test_pybedtools.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.09s ===============================



š„‚ src/bin/tests/test_pybedtools.py failed.
Copy code
(virtualenv3) mpcusack@mpcp15:~/dev/color$ python
Python 3.6.15 (default, Sep 10 2021, 00:26:58) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pybedtools
>>>
h
Hmm, Google tells me that 
undefined symbol PyFPE_jbuf
is a fairly ubiquitous problem in several contexts, unrelated to Pants. So there is some difference in environments between the Pants case and the raw case.
Is this the exact same interpreter? (e.g., can you try 
/home/mpcusack/.pyenv/versions/3.6.15/python -c "import pybedtools"
)
And, can you try 
import fpectl
in both cases? (Pants test and raw interpreter)?
Discussions like this one lead me to believe that this is due to building the pybedtools sdist on an interpreter with fpectl and then running it on one without fpectl.
e
Hi Benjy. Thanks for looking at this. I think it's something to do with the way pants/pex is building/installing the module, but I'm not sure where.
Copy code
mpcusack@mpcp15:~$ pyenv activate pybedtoolstest
(pybedtoolstest) mpcusack@mpcp15:~$ python -V
Python 3.6.15
(pybedtoolstest) mpcusack@mpcp15:~$ pip install pybedtools==0.8.2
Collecting pybedtools
  Using cached <https://files.pythonhosted.org/packages/5b/57/46e9c8135954550d1d3a05db5b0462a264239f7f71c78af304bb47c5c900/pybedtools-0.8.2.tar.gz>
Collecting six (from pybedtools)
  Using cached <https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl>
Collecting pysam (from pybedtools)
  Using cached <https://files.pythonhosted.org/packages/03/ef/6e7342dbe0571fa078bd312f07dcb9b0693b89f8f6fbdf7876e4423374b9/pysam-0.17.0-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.whl>
Installing collected packages: six, pysam, pybedtools
  Running setup.py install for pybedtools ... done
Successfully installed pybedtools-0.8.2 pysam-0.17.0 six-1.16.0
(pybedtoolstest) mpcusack@mpcp15:~$ python -c "import pybedtools"
(pybedtoolstest) mpcusack@mpcp15:~$ echo $?
0
h
Can you try the 
import fpectl
thing in each environment?
e
For fpectl: Trying to import from the pyenv python 3.6.15 virtual env.
Copy code
(pybedtoolstest) mpcusack@mpcp15:~$ python -c "import fpectl"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'fpectl'
From pants:
Copy code
mpcusack@mpcp15:~/dev/color$ ./pants test   src/bin/tests/test_fpectl.py 
19:50:22.53 [WARN] No `.python-version` file found in the build root, but <PYENV_LOCAL> was set in `[python-setup].interpreter_search_paths`.
19:50:24.65 [INFO] Completed: Building requirements.pex
19:50:27.70 [INFO] Completed: Building pytest.pex from pytest_default_lockfile.txt
19:50:28.37 [INFO] Completed: Building pytest_runner.pex
19:50:28.75 [ERROR] Completed: test - src/bin/tests/test_fpectl.py failed (exit code 2).
============================= test session starts ==============================
platform linux -- Python 3.6.15, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
rootdir: /tmp/process-executionxWqBky, configfile: pyproject.toml
plugins: cov-3.0.0
collected 0 items / 1 error

==================================== ERRORS ====================================
________________ ERROR collecting src/bin/tests/test_fpectl.py _________________
ImportError while importing test module '/tmp/process-executionxWqBky/src/bin/tests/test_fpectl.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/home/mpcusack/.pyenv/versions/3.6.15/lib/python3.6/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
src/bin/tests/test_fpectl.py:1: in <module>
    import fpectl
E   ModuleNotFoundError: No module named 'fpectl'
---- generated xml file: /tmp/process-executionxWqBky/var/pytest-report.xml ----
=========================== short test summary info ============================
ERROR src/bin/tests/test_fpectl.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.08s ===============================



š„‚ src/bin/tests/test_fpectl.py failed.
mpcusack@mpcp15:~/dev/color$ cat src/bin/tests/test_fpectl.py
import fpectl


def test_fpectl():
    ...
I'm actually unable to get the 
fpectl
module to be accessible no matter what I do
Copy code
mpcusack@mpcp15:~/dev/color$ PYTHON_CONFIGURE_OPTS="--with-fpectl" pyenv install 3.6.15
pyenv: /home/mpcusack/.pyenv/versions/3.6.15 already exists
continue with installation? (y/N) y
Installing Python-3.6.15...
Installed Python-3.6.15 to /home/mpcusack/.pyenv/versions/3.6.15

mpcusack@mpcp15:~/dev/color$ python -c "import fpectl"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'fpectl'
However now that I have built python 3.6.15 with 
--with-fpectl
I can use 
pybedtools
from pants
Copy code
āœ“ src/bin/tests/test_pybedtools.py succeeded.
š„‚ src/bin/tests/test_fpectl.py failed.
The original 3.6.15, without 
--with-fpectl
, virtualenv with 
pybedtools
pip installed worked though, could pants be selecting the wrong wheel to install?
h
AFAICT 
pybedtools
is distributed only as an sdist, so the problem is related to building wheels rather than selecting existing ones (and in any case I don't think there is an environment marker to denote the presence/absence of fpectl?)
It's good that things are working now, but this is still a bit of a heisenbug. I don't think this is related specifically to Pants, but more to the fact that you had on your system one interpreter with fpectl and one without, and happened to build and cache a wheel on one and then use it on the other.
E.g., possibly system interpreter vs pyenv interpreter
So I think to make the solution robust it might make sense to pick one destiny (with fpectl or without, and I think without is considered safer), ensure that all interpreters pants can discover conform to that, then nuke 
~/.cache/pants/named_caches/pex_root/venvs
and rebuild all the wheels.
E.g., one way to ensure that all interpreters pants can discover are conformant is to set 
interpreter_search_path
to 
"['PYENV']"
, so it will only consider pyenv interpreters. I don't know if that is a change you can make globally though.
Were you encountering this problem just on one machine, or was this widespread? If it was just one machine, and you have a fix, maybe that's sufficient.
Anyway, that is my best understanding of the problem. Arguably you may want to get this working without fpectl, or maybe you're ok as-is šŸ™‚
e
So far just from my machine, but I will see of others can replicate it. FWIW our interpreter_search_path does not include PATH. It only includes ASDF and PYENV, and I only use 
pyenv
. I only have 2 pythons installed via pyenv (3.6.15 & 3.9.7), both of which were (originally) installed without any special env vars/flags.
So make things extra confusing I no longer can replicate the failure. I reinstalled 3.6.15 w/o the 
--with-fpectl
and it continues to work....
h
Hmmm, and Pants uses its own pip cache, so you shouldn't be contaminated by outside pip runs
Very curious
5 Views
Open in Slack
PreviousNext
Powered by