lemon-oxygen-72498
11/23/2022, 5:03 PM.
└── libs
├── a
│ ├── BUILD
│ ├── kaiko
│ │ └── a <- *.py files in there
│ └── requirements.txt
├── b
│ ├── BUILD
│ └── kaiko
│ └── b <- *.py files in there
└── c
├── BUILD
└── kaiko
└── c <- *.py files in there
So in the library a
- that depends on both b
and c
- we have imports of the from from kaiko import b
and from kaiko import c
, because kaiko
is a shared namespace. Both libs/a
, libs/b
, and libs/c
are roots.
`a`'s requirements.txt
contains editable installs stanzas (-e ../b
and -e ../c
).
That doesn't work when typechecking with mypy
(and pyright
), I get Cannot find implementation or library stub for module named "kaiko"
and Cannot find implementation or library stub for module named "kaiko.b"
errors 😞💥 🏴☠️ (depending on the import's shape).
Should I use modules and module mapping statements in my BUILD
file to circumvent this? I clearly see that my setup seems not to fit w.r.t. how pants treats imports, but I don't understand how I would write the mappings.bitter-ability-32190
11/23/2022, 5:04 PMkaiko
is an implicit namepsace package?lemon-oxygen-72498
11/23/2022, 5:05 PMpackages = [
{ include = "kaiko" }
]
bitter-ability-32190
11/23/2022, 5:05 PMlemon-oxygen-72498
11/23/2022, 5:06 PMYou can try making it an explicit namespace packageWhat does this entail exactly?
bitter-ability-32190
11/23/2022, 5:06 PMThefile for the namespace package needs to contain only the following:__init__.py
Copy code__path__ = __import__('pkgutil').extend_path(__path__, __name__)
lemon-oxygen-72498
11/23/2022, 5:09 PM__init__.py
files already contain things 🤔bitter-ability-32190
11/23/2022, 5:10 PM__init__.py
file immediately underneath any kaiko
directoryhappy-kitchen-89482
11/23/2022, 5:11 PMkaiko
is a namespace package, if any of those kaiko
dirs has an __init__.py
then they all must, and it must contain that line Josh postedbitter-ability-32190
11/23/2022, 5:14 PMif none of them do, then it should work as an implicit namespaceThis trips up
mypy
though 😕lemon-oxygen-72498
11/23/2022, 7:24 PMlibs/a/kaiko
directories don't have a ___init___.py
file (libs/a/kaiko/a
does).libs
structure I have outlined in my message on #general, bringing in libs/{b,c}/kaiko/{b,c}/**.py
when typechecking a
.happy-kitchen-89482
11/23/2022, 8:52 PMbitter-ability-32190
11/23/2022, 10:27 PMlemon-oxygen-72498
11/24/2022, 3:01 PM> git clone <https://github.com/smelc/pants-implicit-namespaces-typecheck>
> cd pants-implicit-namespaces-typecheck
> PANTS_SHA=a4054868c11690874005999cb80100e4da107538 ./pants check ::
bitter-ability-32190
11/24/2022, 3:20 PMlemon-oxygen-72498
11/29/2022, 10:10 AMpyright
work in the repo https://github.com/smelc/pants-implicit-namespaces-typecheck (see the detailed description in the repo's README.md), that would be tremendously helpful to me. cc @wide-midnight-78598wide-midnight-78598
11/29/2022, 11:28 AMI have made a minimal example:Oh 5 days ago? Sorry - I missed this message entirely
lemon-oxygen-72498
11/29/2022, 11:36 AMwide-midnight-78598
11/29/2022, 11:41 AMnpx pyright@1.1.258 .
No configuration file found.
No pyproject.toml file found.
stubPath /Users/sj/Developer/scratch/pants-implicit-namespaces-typecheck/typings is not a valid directory.
Assuming Python platform Darwin
Searching for source files
Found 3 source files
pyright 1.1.258
/Users/sj/Developer/scratch/pants-implicit-namespaces-typecheck/libs/a/mycorp/a/a.py
/Users/sj/Developer/scratch/pants-implicit-namespaces-typecheck/libs/a/mycorp/a/a.py:1:20 - error: "b" is unknown import symbol (reportGeneralTypeIssues)
1 error, 0 warnings, 0 informations
Completed in 0.526sec
So, while the error isn't great, at least pyright
in and out of pants react the samelemon-oxygen-72498
11/29/2022, 11:44 AMpants
😐 I don't have have it.
Did you run pip install -r requirements.txt
in libs/a
before running pyright
? (doing so will install -e ../b
)wide-midnight-78598
11/29/2022, 12:30 PMlibs/a/requirements.txt
? While the pants.toml
is at the root. As Benjy mentioned above, "Pants will ignore editable installs in requirements.txt", so that should be ignored.
Now, the question is why Pants isn't picking it up itself via dep inference
Source roots are:
PANTS_SHA=a4054868c11690874005999cb80100e4da107538 ./pants roots
.
libs/a
libs/b
Which I believe should be correct.
Checking out all our targets:
PANTS_SHA=a4054868c11690874005999cb80100e4da107538 ./pants list :: ⎇ main*
libs/a:mycorp-a
libs/a/mycorp/a/a.py:../../mycorp-a
libs/a/mycorp/a:a
libs/a/mycorp/a/a.py
libs/b:mycorp-b
libs/b/mycorp/b/__init__.py:../../mycorp-b
libs/b/mycorp/b/b.py:../../mycorp-b
libs/b/mycorp/b:b
libs/b/mycorp/b/__init__.py
libs/b/mycorp/b/b.py
mycorp/a/
doesn't have an initpy - should it? https://github.com/smelc/pants-implicit-namespaces-typecheck/tree/main/libs/a/mycorp/a
The weird part for me is that there are two sets of BUILD files referring to the same files:
https://github.com/smelc/pants-implicit-namespaces-typecheck/blob/main/libs/a/BUILD
https://github.com/smelc/pants-implicit-namespaces-typecheck/blob/main/libs/a/mycorp/a/BUILD
Ditto for B, so removing one of those should help with the warnings
Transitive deps on a.py
PANTS_SHA=a4054868c11690874005999cb80100e4da107538 ./pants dependencies --transitive libs/a/mycorp/a/a.py
libs/b/mycorp/b/__init__.py:../../mycorp-b
Transitive deps on a.py after explicitly adding dependencies=["libs/b:mycorp-b"]
to A's BUILD
PANTS_SHA=a4054868c11690874005999cb80100e4da107538 ./pants dependencies --transitive libs/a/mycorp/a/a.py
libs/b/mycorp/b/__init__.py:../../mycorp-b
libs/b/mycorp/b/b.py:../../mycorp-b
lemon-oxygen-72498
11/29/2022, 2:45 PMthe requirements is an editable install fromYes correct (this is a development setup)libs/a/requirements.txt
wide-midnight-78598
11/29/2022, 2:47 PMmypy
also fails on this - pyright
plugin shares a bunch of mypy
code, so its a good check (this might be a configuration problem though, with implicit vs explicit namespaces, I always have to read the docs for that):
09:47:12.81 [INFO] Completed: Building mypy.pex from mypy_default.lock
09:47:14.00 [INFO] Completed: Building requirements_venv.pex
09:47:23.33 [ERROR] Completed: Typecheck using MyPy - mypy - mypy failed (exit code 1).
libs/a/mycorp/a/a.py:1: error: Cannot find implementation or library stub for module named "mycorp"
libs/a/mycorp/a/a.py:1: note: See <https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports>
Found 1 error in 1 file (checked 3 source files)
mycorp
to mycorp1
to see what error is spit out: libs/a/mycorp/a/a.py:1:6 - error: Import "mycorp1" could not be resolved (reportMissingImports)
Okay, so a few things to get pyright
working here: I think the constant uses of a
and b
really messed me up - so this took a while:
from b.mycorp.b import b
def add3(x: int, y: int, z: int) -> int:
return b.add(x, y) + z
It looks like there is a b
module in the b
package of mycorp
which is in b
from the perspective of Pants's Pyright.libs/b
shouldn't that first b
package be dropped?pyright
but my guess is that there are some more configurations that need to be passed in via the pyrightconfig.json or pyproject.toml{
"include": [
"libs/a",
"libs/b",
"."
]
}
Doesn't work on it's own, but hopefully in conjunction with other settings - or maybe extraPaths
- that appears to work.
Will open a ticket for finally getting pyright
configurations handled in the tool. Some parts of config need to be overridden, so it's a bit messylemon-oxygen-72498
11/29/2022, 8:25 PMpyright
was picking something up from pyproject.toml
, but pyright's documentation states that it only inspects things under the [pyright]
entry and I have nothing namespace specific in my real life scenario, my [pyright]
section is as follows:
[tool.pyright]
reportMissingParameterType = true
strictListInference = true
strictDictionaryInference = true
strictSetInference = true
pyproject.toml
in `pants`' sandox, and running __run.sh
manually, thinking it could maybe pick some stuff from [tool.poetry]
(such as packages = [ { include = mycorp" } ]
) but I couldn't trigger a change in `pyright`'s behavior when doing so 🤷pyright
indeed only reads the [pyright]
entry from pyproject.toml
🙂wide-midnight-78598
11/29/2022, 8:36 PMvenv
and the extraPaths
(source roots). Makes your sample repo pass.
I have some cleanup, docs, and re-factoring, but high-level seems to be finelemon-oxygen-72498
12/11/2022, 2:26 PMwide-midnight-78598
12/11/2022, 2:27 PMlemon-oxygen-72498
12/11/2022, 2:28 PMwide-midnight-78598
12/14/2022, 2:01 PM./pants_from_sources roots
lemon-oxygen-72498
12/14/2022, 4:44 PM→ ./pants_from_sources roots
.
libs/a
libs/b
and a correct list on my production repo.
In both cases I have marker_filenames = ["pyproject.toml"]
.wide-midnight-78598
12/14/2022, 4:48 PMlemon-oxygen-72498
12/14/2022, 4:56 PMgit diff
is clean, source pants
repo is at the correct SHA, etc. 🤷 Would it be useful if I zip the generated sandbox when it fails? (so that you compare with your sandbox)node-v16.15.0-linux-x64
folder from it, to save space)wide-midnight-78598
12/15/2022, 2:09 AMextraPaths
isn't there is a surprise. I'd wager that the wrong version is used or something