https://pantsbuild.org/ logo
#general
Title
# general
p

polite-angle-19513

08/25/2021, 3:49 PM
Hello all, thanks to all for previous help. Have a monorepo. Have finally given up trying to fit Pants with PEP420 implied namespace pacakges. I'm using the pkgutil namespace package with the magic init.py under each '*hankai*' namespace that contains:
__path__ = __import__("pkgutil").extend_path(__path__, __name__)  # type: ignore
However, in a simplified pants-test setup I cannot pants to do what I think it ought too. It's been a very frustrating and quite time consuming attempt to adopt Pants. I feel like I'm so close to a working setup and then no matter what I do it seems to work as expected in many ways and then falls over in others. (And vice versa when I setup with PEP420; mypy works in all and pylint fails - opposite to the pkgutil namespace setup). The Pants docs, while referencing namespace packages, wasn't enough for me to glean the best way. Maybe another monorepo user might have some wisdom on the matter? (TY) . ├── mypy.ini ├── pants ├── pants.toml ├── pylintrc ├── pyproject.toml └── src └── python ├── hankai-dep (depends hankai-util) │ ├── BUILD │ ├── hankai │ │ ├── init.py │ │ └── dep │ │ ├── init.py │ │ └── dep.py │ ├── poetry.lock │ ├── poetry.toml │ ├── pyproject.toml │ └── requirements.txt └── hankai-util (independent) ├── BUILD ├── hankai │ ├── init.py │ └── util │ ├── init.py │ └── util.py ├── poetry.lock ├── poetry.toml ├── pyproject.toml └── requirements.txt hankai-util is independent, hankak-dep is dependent on hankai-util src/python/hankai-util/BUILD python_requirements() python_library( name="lib", dependencies=[ ":loguru", ":pydantic", ], sources=[ "hankai/**/*.py", ] ) src/python/hankai-dep/BUILD python_requirements() python_library( name="lib", dependencies=[ ":loguru", ":pydantic", "src/python/hankai-util:lib", ], sources=[ "hankai/**/*.py", ] ) ---------------------- LIST ./pants list :: src/python/hankai-dep:colorama src/python/hankai-dep:dist src/python/hankai-dep:lib src/python/hankai-dep:loguru src/python/hankai-dep:pydantic src/python/hankai-dep:requirements.txt src/python/hankai-dep:typing-extensions src/python/hankai-dep:win32-setctime src/python/hankai-util:colorama src/python/hankai-util:dist src/python/hankai-util:lib src/python/hankai-util:loguru src/python/hankai-util:pydantic src/python/hankai-util:requirements.txt src/python/hankai-util:tqdm src/python/hankai-util:typing-extensions src/python/hankai-util:win32-setctime ---------------------- DEPENDENCIES/DEPENDEES ./pants dependencies :: src/python/hankai-dep/hankai/__init__.py:../lib src/python/hankai-dep/hankai/dep/__init__.py:../../lib src/python/hankai-dep/hankai/dep/dep.py:../../lib src/python/hankai-dep:loguru src/python/hankai-dep:pydantic src/python/hankai-dep:requirements.txt src/python/hankai-util/hankai/__init__.py:../lib src/python/hankai-util/hankai/util/__init__.py:../../lib src/python/hankai-util/hankai/util/util.py:../../lib src/python/hankai-util:dist src/python/hankai-util:loguru src/python/hankai-util:pydantic src/python/hankai-util:requirements.txt ./pants dependencies src/python/hankai-_dep_:lib src/python/hankai-dep/hankai/__init__.py:../lib src/python/hankai-dep/hankai/dep/__init__.py:../../lib src/python/hankai-dep/hankai/dep/dep.py:../../lib src/python/hankai-dep:loguru src/python/hankai-dep:pydantic src/python/hankai-util/hankai/__init__.py:../lib src/python/hankai-util/hankai/util/__init__.py:../../lib src/python/hankai-util/hankai/util/util.py:../../lib ./pants dependencies src/python/hankai-util:lib src/python/hankai-util/hankai/__init__.py:../lib src/python/hankai-util/hankai/util/__init__.py:../../lib src/python/hankai-util/hankai/util/util.py:../../lib src/python/hankai-util:loguru src/python/hankai-util:pydantic ./pants dependees src/python/hankai-util:lib src/python/hankai-dep:lib src/python/hankai-dep/hankai/__init__.py:../lib src/python/hankai-dep/hankai/dep/__init__.py:../../lib src/python/hankai-dep/hankai/dep/dep.py:../../lib src/python/hankai-util:dist ---------------------- LINT *./pants lint :: (*Successful) ./pants lint src/python/hankai-util:lib (Successful) ./pants lint src/python/hankai-dep:lib (FAILS) - Expected that the hankai-util deps would be referenced since ./pants knows about them as shown above. src/python/hankai-dep/hankai/dep/dep.py20: E0611: No name 'util' in module 'hankai' (no-name-in-module) src/python/hankai-dep/hankai/dep/dep.py20: E0401: Unable to import 'hankai.util' (import-error) src/python/hankai-dep/hankai/dep/dep.py158: E1101: Module 'hankai' has no 'util' member (no-member) ./pants lint src/python/hankai-dep:lib src/python/hankai-util:lib (Successful) - But I expected the explicit dependency to be used and to not have to directly reference it. ---------------------- TYPECHECK ./pants typecheck :: (FAILS) - I'm not doing PEP420. Using pkgutil namespace packaging. How am I supposed to use pants with any form of namespace package support?? src/python/hankai-util/hankai/__init__.py: error: Duplicate module named "hankai" (also at "src/python/hankai-dep/hankai/__init__.py") src/python/hankai-util/hankai/__init__.py: note: Are you missing an init.py? Alternatively, consider using --exclude to avoid checking one of them. Found 1 error in 1 file (errors prevented further checking) ./pants typecheck src/python/hankai-util:lib (Successful) ./pants typecheck src/python/hankai-dep:lib (FAILS) - Expected the hankai-util deps would be referenced since ./pants knows about them as shown above. src/python/hankai-dep/hankai/dep/dep.py21: error: Cannot find implementation or library stub for module named "hankai.util" [import] import hankai.util ^ src/python/hankai-dep/hankai/dep/dep.py21: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports Found 1 error in 1 file (checked 3 source files) ./pants typecheck src/python/hankai-dep:lib src/python/hankai-util:lib (FAILS) as before with :: src/python/hankai-util/hankai/__init__.py: error: Duplicate module named "hankai" (also at "src/python/hankai-dep/hankai/__init__.py") src/python/hankai-util/hankai/__init__.py: note: Are you missing an init.py? Alternatively, consider using --exclude to avoid checking one of them. Found 1 error in 1 file (errors prevented further checking) ============================================ I feel like I'm so close to a working but am just not getting there. Maybe another monorepo user might have some wisdom on the matter? Thank you all for your consideration and help.
w

witty-crayon-22786

08/25/2021, 3:54 PM
hey James: sorry that this has been troublesome.
do you mind if i lift this into a Github issue? i think that that might help us to grok it
👍 2
opened https://github.com/pantsbuild/pants/issues/12650 for this: would welcome assistance!
1
@polite-angle-19513: one other relevant bit I can add to the ticket: what’s the output of
./pants roots
?
p

polite-angle-19513

08/25/2021, 4:07 PM
./pants roots src/python [source] roots = [ "src/python", ] I'd also tried with [source] marker_filenames = [ "pyproject.toml", ]
w

witty-crayon-22786

08/25/2021, 4:08 PM
yea, those roots won’t work: each source root is like a PYTHONPATH entry, and you only have one:
src/python
if you chop
src/python
off the front of those directories, you’ll get
import
statements like
import hankai-util.hankai.util
etc
so yea: you will need something like the
marker_filenames
strategy, because you need multiple source roots.
this page explains: https://www.pantsbuild.org/docs/source-roots … but because you have multiple top-level projects, your goal is to have 1 source root per project
@polite-angle-19513: ok, thanks! i’ll respond on the ticket
p

polite-angle-19513

08/25/2021, 4:13 PM
Just put back to marker_filenames - and works... ./pants roots . src src/python src/python/hankai-dep src/python/hankai-util My apologies - I've put in and taken out so many settings all over to see what works that I've shot myself in the foot.
w

witty-crayon-22786

08/25/2021, 4:14 PM
great… you might have a few more than you really need there, but that looks like a step in the right direction
it looks like you also have a poetry.toml in each project directory, so that might be a better marker file
👍 1
5 Views