#17919 pants test fails to include source root on path when python_tests depends on grandparent python_sources Issue created by cburroughs Describe the bug I'm working with a pre-existing pattern where python tests are in a test directory (I think this is pretty typical!) and there is also a utility file located in the test directory that is a dependence of the tests and not a test itself (this seems debatable as a practice, but I think should work). I'm seeing different behavior with

sys.path

when the utility class is in a parent vs-grandparent directory of the tests.

$ tree test/
test/
├── BUILD
├── __init__.py
├── test_util.py <--- utility class, not actually a test
└── unit
    ├── BUILD
    ├── __init__.py
    ├── scripts
    │   ├── BUILD
    │   └── test_some_more_stuff.py <-- this target can *not* import test.test_util
    └── test_some_stuff.py <-- this target can import test.test_util

Notably the root of the sandbox is the first item in

sys.path

in the "parent case" but is not explicitly on the path at all in the "grandparent" case. This means`import test` picks up the python stdlib tests (eg

import test.test_cgi

"works") As a work around one can add:

So, this works if I add
$ cat pytest.ini 
[pytest]
pythonpath=.

but pants ought to be able to figure all of this out on its own.

$ ./pants roots
18:00:04.23 [INFO] Initialization options changed: reinitializing scheduler...
18:00:04.96 [INFO] Scheduler initialized.
.

$ cat test/unit/test_some_stuff.py 
import sys

from test.test_util import SpiffyUtil

import scripts.hello

def test_hi():
    print(sys.path)
    assert False

$ cat test/unit/scripts/test_some_more_stuff.py 
import sys
print(sys.path)    

from test.test_util import SpiffyUtil

import scripts.hello

def test_hi():
    assert False

$ ./pants --keep-sandboxes=on_failure test --output=all --force test/unit/test_some_stuff.py 
12:35:04.49 [INFO] Preserving local process execution dir /tmp/pants-sandbox-qvTKNh for Run Pytest for test/unit/test_some_stuff.py:tests
12:35:04.49 [ERROR] Completed: Run Pytest - test/unit/test_some_stuff.py:tests - failed (exit code 1).
============================= test session starts ==============================
platform linux -- Python 3.7.15, pytest-7.0.1, pluggy-1.0.0
rootdir: /tmp/pants-sandbox-qvTKNh
plugins: xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 1 item

test/unit/test_some_stuff.py F                                           [100%]

=================================== FAILURES ===================================
___________________________________ test_hi ____________________________________

   def test_hi():
       print(sys.path)
>       assert False
E       assert False

test/unit/test_some_stuff.py:9: AssertionError
----------------------------- Captured stdout call -----------------------------
['/tmp/pants-sandbox-qvTKNh', '/home/ecsb/.cache/pants/named_caches/pex_root/venvs/7d2c8b5e85a93a776d27d91c56ed99e28f9ccd69/49ea33172fd4069e547596c7f14f1329f27e25eb', '/usr/local/lib/python37.zip', '/usr/local/lib/python3.7', '/usr/local/lib/python3.7/lib-dynload', '/home/ecsb/.cache/pants/named_caches/pex_root/venvs/s/731fd73e/venv/lib/python3.7/site-packages', '.', '.']
- generated xml file: /tmp/pants-sandbox-qvTKNh/test.unit.test_some_stuff.py.tests.xml -
=========================== short test summary info ============================
FAILED test/unit/test_some_stuff.py::test_hi - assert False

$ ./pants --keep-sandboxes=on_failure test test/unit/scripts/test_some_more_stuff.py 
12:35:28.22 [INFO] Preserving local process execution dir /tmp/pants-sandbox-MfHqjH for Run Pytest for test/unit/scripts/test_some_more_stuff.py:tests
12:35:28.22 [ERROR] Completed: Run Pytest - test/unit/scripts/test_some_more_stuff.py:tests - failed (exit code 2).
============================= test session starts ==============================
platform linux -- Python 3.7.15, pytest-7.0.1, pluggy-1.0.0
rootdir: /tmp/pants-sandbox-MfHqjH
plugins: xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 0 items / 1 error

==================================== ERRORS ====================================
__________ ERROR collecting test/unit/scripts/test_some_more_stuff.py __________
ImportError while importing test module '/tmp/pants-sandbox-MfHqjH/test/unit/scripts/test_some_more_stuff.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/local/lib/python3.7/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
test/unit/scripts/test_some_more_stuff.py:4: in <module>
    from test.test_util import SpiffyUtil
E   ModuleNotFoundError: No module named 'test.test_util'
------------------------------- Captured stdout --------------------------------
['/tmp/pants-sandbox-MfHqjH/test/unit/scripts', '/home/ecsb/.cache/pants/named_caches/pex_root/venvs/7d2c8b5e85a93a776d27d91c56ed99e28f9ccd69/49ea33172fd4069e547596c7f14f1329f27e25eb', '/usr/local/lib/python37.zip', '/usr/local/lib/python3.7', '/usr/local/lib/python3.7/lib-dynload', '/home/ecsb/.cache/pants/named_caches/pex_root/venvs/s/731fd73e/venv/lib/python3.7/site-packages', '.', '.']
- generated xml file: /tmp/pants-sandbox-MfHqjH/test.unit.scripts.test_some_more_stuff.py.tests.xml -
=========================== short test summary info ============================
ERROR test/unit/scripts/test_some_more_stuff.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.12s ===============================



✕ test/unit/scripts/test_some_more_stuff.py:tests failed in 0.40s.

$ find test -iname BUILD | xargs -t cat
cat test/BUILD test/unit/BUILD test/unit/scripts/BUILD
python_tests(
    name="tests",
    sources=['!test_utils.py']
)

python_sources(
    name='test_util',
    sources=['__init__.py', 'test_util.py'])

    
python_tests(
    name="tests",
)
python_tests(
    name="tests",
)

Pants version

2.16.0.dev4

OS Linux Additional info Slack 🧵 https://pantsbuild.slack.com/archives/C046T6T9U/p1672767253531889 pantsbuild/pants