https://pantsbuild.org/ logo
#general
Title
# general
c

curved-manchester-66006

01/03/2023, 5:34 PM
I'm working with a pre-existing pattern where python tests are in a
test
directory (I think this is pretty typical!) and there is also a utility file located in the test directory that is a dependence of the tests and not a test itself (this seems debatable as a practice, but I think should work?). I'm seeing different behavior with
sys.path
when the utility class is in a parent vs-grandparent directory of the tests.
Copy code
$ tree test/
test/
├── BUILD
├── __init__.py
├── test_util.py <--- utility class, not actually a test
└── unit
    ├── BUILD
    ├── __init__.py
    ├── scripts
    │   ├── BUILD
    │   └── test_some_more_stuff.py <-- this target can *not* import test.test_util
    └── test_some_stuff.py <-- this target can import test.test_util
🧵
Copy code
$ cat test/unit/test_some_stuff.py 
import sys

from test.test_util import SpiffyUtil

import scripts.hello

def test_hi():
    print(sys.path)
    assert False
Copy code
$ cat test/unit/scripts/test_some_more_stuff.py 
import sys
print(sys.path)    

from test.test_util import SpiffyUtil

import scripts.hello

def test_hi():
    assert False
Copy code
$ ./pants --keep-sandboxes=on_failure test --output=all --force test/unit/test_some_stuff.py 
12:35:04.49 [INFO] Preserving local process execution dir /tmp/pants-sandbox-qvTKNh for Run Pytest for test/unit/test_some_stuff.py:tests
12:35:04.49 [ERROR] Completed: Run Pytest - test/unit/test_some_stuff.py:tests - failed (exit code 1).
============================= test session starts ==============================
platform linux -- Python 3.7.15, pytest-7.0.1, pluggy-1.0.0
rootdir: /tmp/pants-sandbox-qvTKNh
plugins: xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 1 item

test/unit/test_some_stuff.py F                                           [100%]

=================================== FAILURES ===================================
___________________________________ test_hi ____________________________________

    def test_hi():
        print(sys.path)
>       assert False
E       assert False

test/unit/test_some_stuff.py:9: AssertionError
----------------------------- Captured stdout call -----------------------------
['/tmp/pants-sandbox-qvTKNh', '/home/ecsb/.cache/pants/named_caches/pex_root/venvs/7d2c8b5e85a93a776d27d91c56ed99e28f9ccd69/49ea33172fd4069e547596c7f14f1329f27e25eb', '/usr/local/lib/python37.zip', '/usr/local/lib/python3.7', '/usr/local/lib/python3.7/lib-dynload', '/home/ecsb/.cache/pants/named_caches/pex_root/venvs/s/731fd73e/venv/lib/python3.7/site-packages', '.', '.']
- generated xml file: /tmp/pants-sandbox-qvTKNh/test.unit.test_some_stuff.py.tests.xml -
=========================== short test summary info ============================
FAILED test/unit/test_some_stuff.py::test_hi - assert False
Copy code
$ ./pants --keep-sandboxes=on_failure test test/unit/scripts/test_some_more_stuff.py 
12:35:28.22 [INFO] Preserving local process execution dir /tmp/pants-sandbox-MfHqjH for Run Pytest for test/unit/scripts/test_some_more_stuff.py:tests
12:35:28.22 [ERROR] Completed: Run Pytest - test/unit/scripts/test_some_more_stuff.py:tests - failed (exit code 2).
============================= test session starts ==============================
platform linux -- Python 3.7.15, pytest-7.0.1, pluggy-1.0.0
rootdir: /tmp/pants-sandbox-MfHqjH
plugins: xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 0 items / 1 error

==================================== ERRORS ====================================
__________ ERROR collecting test/unit/scripts/test_some_more_stuff.py __________
ImportError while importing test module '/tmp/pants-sandbox-MfHqjH/test/unit/scripts/test_some_more_stuff.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/local/lib/python3.7/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
test/unit/scripts/test_some_more_stuff.py:4: in <module>
    from test.test_util import SpiffyUtil
E   ModuleNotFoundError: No module named 'test.test_util'
------------------------------- Captured stdout --------------------------------
['/tmp/pants-sandbox-MfHqjH/test/unit/scripts', '/home/ecsb/.cache/pants/named_caches/pex_root/venvs/7d2c8b5e85a93a776d27d91c56ed99e28f9ccd69/49ea33172fd4069e547596c7f14f1329f27e25eb', '/usr/local/lib/python37.zip', '/usr/local/lib/python3.7', '/usr/local/lib/python3.7/lib-dynload', '/home/ecsb/.cache/pants/named_caches/pex_root/venvs/s/731fd73e/venv/lib/python3.7/site-packages', '.', '.']
- generated xml file: /tmp/pants-sandbox-MfHqjH/test.unit.scripts.test_some_more_stuff.py.tests.xml -
=========================== short test summary info ============================
ERROR test/unit/scripts/test_some_more_stuff.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.12s ===============================



✕ test/unit/scripts/test_some_more_stuff.py:tests failed in 0.40s.
Notably the root of the sandbox is the first item in
sys.path
in the "parent case" but is not on the path at all in the "grandparent" case. This means
import test
picks up the python stdlib tests (eg
import test.test_cgi
)
oh probably relevant:
Copy code
$ find test -iname BUILD | xargs -t cat
cat test/BUILD test/unit/BUILD test/unit/scripts/BUILD
python_tests(
    name="tests",
    sources=['!test_utils.py']
)

python_sources(
    name='test_util',
    sources=['__init__.py', 'test_util.py'])

    
python_tests(
    name="tests",
)
python_tests(
    name="tests",
)
So I got the imports to work(!!!) by adding:
Copy code
So, this works if I add
$ cat pytest.ini 
[pytest]
pythonpath=.
Which seems rather... wonky. I'm not sure if something about this pre-existing file layout is crazy, or if https://www.pantsbuild.org/docs/reference-python_test_utils needs a big "Hey! you probably are going to need this pytest.ini setting!" banner.
w

witty-crayon-22786

01/03/2023, 10:24 PM
is the relevant directory a source root (which will cause it to be placed on the pythonpath)? can check with
./pants roots
c

curved-manchester-66006

01/03/2023, 11:00 PM
Copy code
./pants roots
18:00:04.23 [INFO] Initialization options changed: reinitializing scheduler...
18:00:04.96 [INFO] Scheduler initialized.
.
w

witty-crayon-22786

01/04/2023, 12:06 AM
hm… right. that makes sense. the
pytest.ini
setting shouldn’t be necessary in this case, because Pants should be constructing an appropriate python path for you (consistently containing
.
in this case). would you mind filing a bug containing some of this info? sorry for the trouble!
c

curved-manchester-66006

01/04/2023, 2:02 PM
6 Views