https://pantsbuild.org/ logo
#general
Title
# general
r

rich-london-74860

03/04/2023, 4:02 AM
Hello 👋 I’ve hit a curious issue with unit tests, which I’ve replicated with a toy example. If you clone the repo and run
./pants test ::
then
test/test_module_b.py:module_b
will succeed, but
test/foo/modules/module_a/test_module_a.py
will fail with the error
Copy code
test/foo/dpfs/module_a/test_module_a.py:1: in <module>
    from module_a.dpfs import get_domain, plus_one
E   ModuleNotFoundError: No module named 'module_a.dpfs'
This is strange because
src/bar/dpfs/module_b/__init__.py
also imports
from module_a.dpfs import plus_one
, but the test for that module passes. This also works:
Copy code
$ ./pants repl test/foo/modules/module_a:module_a
Python 3.10.6 (main, Aug 11 2022, 13:49:25) [Clang 13.1.6 (clang-1316.0.21.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from module_a import get_domain, plus_one
>>>
And
./pants check ::
passes Next, if you remove the file
rm test/foo/dpfs/module_a/__init__.py
, then
./pants test ::
will pass for all files, but
./pants check ::
will fail with the error:
Copy code
test/test_module.py: error: Duplicate module named "test_module" (also at "test/foo/modules/module_a/test_module.py")
In summary, it looks like when running the unit-test
test/test_module_b.py:module_b
, the file
test/foo/modules/module_a/__init__.py
will load before
src/foo/modules/module_a/dpfs/__init__.py
because the path
test/foo/dpfs
is added to the front of
sys.path
. This can be fixed by deleting
test/foo/modules/module_a/__init__.py
, but then type-checking fails because of a collision on
test_module.py
. What is the right way out of this mess? Am I starting off on the wrong foot by putting all tests under a parent test directory?
e

enough-analyst-54434

03/04/2023, 4:12 AM
The issue is a general Python one (not Pants specific). You cannot split a package across multiple paths unless you pick one of: + Use no
__init__.py
: See https://peps.python.org/pep-0420/ + Use single line
__init__.py
: See https://packaging.python.org/en/latest/guides/packaging-namespace-packages/#creating-a-namespace-package
In other words, Python doesn't merge duplicate init.py packages unless they all have the same single line magic inside, it picks one winner and disregards all others.
Another factor that plays in are source roots. Make sure
pants roots
prints what you expect.
r

rich-london-74860

03/04/2023, 4:52 AM
Merging packages is not the desired result here. Let me phrase this another way. Why is
test/foo/modules/module_a
treated as package
module_a
instead of
test.foo.modules.module_a
, thereby causing it to collide with
module_a
, the very module it is trying to test, when the source root is
test
not
test/foo/modules
?
e

enough-analyst-54434

03/04/2023, 5:17 AM
Can you provide the output of
pants roots
?
Ah sorry - missed the repo link. Checking that out...
So,
git rm -f test/foo/modules/module_a/__init__.py
plus:
Copy code
$ cat pyproject.toml
[tool.mypy]
namespace_packages = true
explicit_package_bases = true
Fixes.
Copy code
$ pants lint check test ::
21:40:44.92 [INFO] Completed: Format with Black - black made no changes.
21:40:44.92 [INFO] Completed: Format with isort - isort made no changes.
21:40:44.92 [INFO] Completed: Lint using Pylint - pylint succeeded.
Partition: ['CPython>=3.10.*']
************* Module .pylintrc
.pylintrc:1:0: R0022: Useless option value for '--disable', 'bad-continuation' was removed from pylint, see <https://github.com/PyCQA/pylint/pull/3571>. (useless-option-value)
.pylintrc:1:0: W0012: Unknown option value for '--disable', expected a valid pylint message and got 'wrong-input-order' (unknown-option-value)

------------------------------------
Your code has been rated at 10.00/10


PYLINTHOME is now '/home/jsirois/.cache/pylint' but obsolescent '/home/jsirois/.pylint.d' is found; you can safely remove the latter




✓ black succeeded.
✓ isort succeeded.
✓ pylint succeeded.
21:40:44.92 [INFO] Completed: Typecheck using MyPy - mypy - mypy succeeded.
Success: no issues found in 4 source files



✓ mypy succeeded.
21:40:44.93 [INFO] Completed: Run Pytest - test/foo/modules/module_a/test_module.py - succeeded.
============================= test session starts ==============================
platform linux -- Python 3.10.7, pytest-7.0.1, pluggy-1.0.0 -- /home/jsirois/.cache/pants/named_caches/pex_root/venvs/s/0c926e0b/venv/bin/python3.10
cachedir: .pytest_cache
rootdir: /tmp/pants-sandbox-MBWvs2
plugins: forked-1.4.0, cov-3.0.0, xdist-2.5.0
collecting ... collected 2 items

test/foo/modules/module_a/test_module.py::test_get_domain PASSED         [ 50%]
test/foo/modules/module_a/test_module.py::test_plus_one PASSED           [100%]

- generated xml file: /tmp/pants-sandbox-MBWvs2/test.foo.modules.module_a.test_module.py.xml -


============================== 2 passed in 0.61s ===============================


21:40:44.93 [INFO] Completed: Run Pytest - test/test_module.py:module_b - succeeded.
============================= test session starts ==============================
platform linux -- Python 3.10.7, pytest-7.0.1, pluggy-1.0.0 -- /home/jsirois/.cache/pants/named_caches/pex_root/venvs/s/0c926e0b/venv/bin/python3.10
cachedir: .pytest_cache
rootdir: /tmp/pants-sandbox-8iwZM5
plugins: forked-1.4.0, cov-3.0.0, xdist-2.5.0
collecting ... collected 1 item

test/test_module.py::test_plus_y PASSED                                  [100%]

- generated xml file: /tmp/pants-sandbox-8iwZM5/test.test_module.py.module_b.xml -


============================== 1 passed in 0.19s ===============================



✓ test/foo/modules/module_a/test_module.py succeeded in 0.89s (memoized).
✓ test/test_module.py:module_b succeeded in 0.61s (memoized).

Name                                   Stmts   Miss  Cover
----------------------------------------------------------
src/bar/modules/module_b/__init__.py       5      0   100%
src/foo/modules/module_a/__init__.py       7      0   100%
----------------------------------------------------------
TOTAL                                     12      0   100%


Wrote xml coverage report to `dist/coverage/python`
I'll let you read up on those mypy options, but the Pants repo itself has to use them as well.
1
r

rich-london-74860

03/05/2023, 1:10 PM
That worked, thank you!