https://pantsbuild.org/ logo
Join Slack
Powered by
Hello :wave: I’ve hit a curious issue with unit t...
# general
r
Hello 👋 I’ve hit a curious issue with unit tests, which I’ve replicated with a toy example. If you clone the repo and run 
./pants test ::
then 
test/test_module_b.py:module_b
will succeed, but 
test/foo/modules/module_a/test_module_a.py
will fail with the error
Copy code
test/foo/dpfs/module_a/test_module_a.py:1: in <module>
    from module_a.dpfs import get_domain, plus_one
E   ModuleNotFoundError: No module named 'module_a.dpfs'
This is strange because 
src/bar/dpfs/module_b/__init__.py
also imports 
from module_a.dpfs import plus_one
, but the test for that module passes. This also works:
Copy code
$ ./pants repl test/foo/modules/module_a:module_a
Python 3.10.6 (main, Aug 11 2022, 13:49:25) [Clang 13.1.6 (clang-1316.0.21.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from module_a import get_domain, plus_one
>>>
And 
./pants check ::
passes Next, if you remove the file 
rm test/foo/dpfs/module_a/__init__.py
, then 
./pants test ::
will pass for all files, but 
./pants check ::
will fail with the error:
Copy code
test/test_module.py: error: Duplicate module named "test_module" (also at "test/foo/modules/module_a/test_module.py")
In summary, it looks like when running the unit-test 
test/test_module_b.py:module_b
, the file 
test/foo/modules/module_a/__init__.py
will load before 
src/foo/modules/module_a/dpfs/__init__.py
because the path 
test/foo/dpfs
is added to the front of 
sys.path
. This can be fixed by deleting 
test/foo/modules/module_a/__init__.py
, but then type-checking fails because of a collision on 
test_module.py
. What is the right way out of this mess? Am I starting off on the wrong foot by putting all tests under a parent test directory?
e
The issue is a general Python one (not Pants specific). You cannot split a package across multiple paths unless you pick one of: + Use no 
__init__.py
: See https://peps.python.org/pep-0420/ + Use single line 
__init__.py
: See https://packaging.python.org/en/latest/guides/packaging-namespace-packages/#creating-a-namespace-package
In other words, Python doesn't merge duplicate init.py packages unless they all have the same single line magic inside, it picks one winner and disregards all others.
Another factor that plays in are source roots. Make sure 
pants roots
prints what you expect.
r
Merging packages is not the desired result here. Let me phrase this another way. Why is 
test/foo/modules/module_a
treated as package 
module_a
instead of 
test.foo.modules.module_a
, thereby causing it to collide with 
module_a
, the very module it is trying to test, when the source root is 
test
not 
test/foo/modules
?
e
Can you provide the output of 
pants roots
?
Ah sorry - missed the repo link. Checking that out...
So, 
git rm -f test/foo/modules/module_a/__init__.py
plus:
Copy code
$ cat pyproject.toml
[tool.mypy]
namespace_packages = true
explicit_package_bases = true
Fixes.
Copy code
$ pants lint check test ::
21:40:44.92 [INFO] Completed: Format with Black - black made no changes.
21:40:44.92 [INFO] Completed: Format with isort - isort made no changes.
21:40:44.92 [INFO] Completed: Lint using Pylint - pylint succeeded.
Partition: ['CPython>=3.10.*']
************* Module .pylintrc
.pylintrc:1:0: R0022: Useless option value for '--disable', 'bad-continuation' was removed from pylint, see <https://github.com/PyCQA/pylint/pull/3571>. (useless-option-value)
.pylintrc:1:0: W0012: Unknown option value for '--disable', expected a valid pylint message and got 'wrong-input-order' (unknown-option-value)

------------------------------------
Your code has been rated at 10.00/10


PYLINTHOME is now '/home/jsirois/.cache/pylint' but obsolescent '/home/jsirois/.pylint.d' is found; you can safely remove the latter




✓ black succeeded.
✓ isort succeeded.
✓ pylint succeeded.
21:40:44.92 [INFO] Completed: Typecheck using MyPy - mypy - mypy succeeded.
Success: no issues found in 4 source files



✓ mypy succeeded.
21:40:44.93 [INFO] Completed: Run Pytest - test/foo/modules/module_a/test_module.py - succeeded.
============================= test session starts ==============================
platform linux -- Python 3.10.7, pytest-7.0.1, pluggy-1.0.0 -- /home/jsirois/.cache/pants/named_caches/pex_root/venvs/s/0c926e0b/venv/bin/python3.10
cachedir: .pytest_cache
rootdir: /tmp/pants-sandbox-MBWvs2
plugins: forked-1.4.0, cov-3.0.0, xdist-2.5.0
collecting ... collected 2 items

test/foo/modules/module_a/test_module.py::test_get_domain PASSED         [ 50%]
test/foo/modules/module_a/test_module.py::test_plus_one PASSED           [100%]

- generated xml file: /tmp/pants-sandbox-MBWvs2/test.foo.modules.module_a.test_module.py.xml -


============================== 2 passed in 0.61s ===============================


21:40:44.93 [INFO] Completed: Run Pytest - test/test_module.py:module_b - succeeded.
============================= test session starts ==============================
platform linux -- Python 3.10.7, pytest-7.0.1, pluggy-1.0.0 -- /home/jsirois/.cache/pants/named_caches/pex_root/venvs/s/0c926e0b/venv/bin/python3.10
cachedir: .pytest_cache
rootdir: /tmp/pants-sandbox-8iwZM5
plugins: forked-1.4.0, cov-3.0.0, xdist-2.5.0
collecting ... collected 1 item

test/test_module.py::test_plus_y PASSED                                  [100%]

- generated xml file: /tmp/pants-sandbox-8iwZM5/test.test_module.py.module_b.xml -


============================== 1 passed in 0.19s ===============================



✓ test/foo/modules/module_a/test_module.py succeeded in 0.89s (memoized).
✓ test/test_module.py:module_b succeeded in 0.61s (memoized).

Name                                   Stmts   Miss  Cover
----------------------------------------------------------
src/bar/modules/module_b/__init__.py       5      0   100%
src/foo/modules/module_a/__init__.py       7      0   100%
----------------------------------------------------------
TOTAL                                     12      0   100%


Wrote xml coverage report to `dist/coverage/python`
I'll let you read up on those mypy options, but the Pants repo itself has to use them as well.
✅ 1
r
That worked, thank you!
Open in Slack
PreviousNext
Powered by