https://pantsbuild.org/ logo
Powered by
a

average-sugar-68948

12/21/2022, 2:42 PM
Hi folks, I'm getting mypy errors when using protobufs. Anyone have any suggestions? I've enabled the mypy-plugin in the [python-protobuf] section, but still get:
Copy code
./pants check ::
09:37:48.34 [INFO] Completed: Building requirements.pex with 5 requirements: bitarray==2.5.1, numpy==1.23.4, protobuf==4.21.12, simpy==4.0.1, types-protobuf==4.21.0.2
09:37:48.75 [INFO] Completed: Building requirements_venv.pex
09:37:49.18 [ERROR] Completed: Typecheck using MyPy - mypy failed (exit code 1).
fabric/proto/v1/workload_pb2.pyi:6: error: Library stubs not installed for "google.protobuf.descriptor" (or incompatible with Python 3.7)
fabric/proto/v1/workload_pb2.pyi:6: note: Hint: "python3 -m pip install types-protobuf"
fabric/proto/v1/workload_pb2.pyi:6: note: (or run "mypy --install-types" to install all missing stub packages)
...
As you can see, I tried explicitly adding the types-protobuf dependency. I also tried the recommended --install-types in mypy args, but it doesn't have an auto-yes feature so can't progress. Trying to pass --exclude to mypy args doesn't work when you're in file-list mode (which is what pants seems to do)
e

enough-analyst-54434

12/21/2022, 3:08 PM
So you're saying we can safely ignore the Python 3.7 warning there? You know that's not it for unspecified reasons; so it must be something else?
a

average-sugar-68948

12/21/2022, 3:09 PM
actually that's weird, I'm not sure where the 3.7 is coming from. My python is 3.9
e

enough-analyst-54434

12/21/2022, 3:13 PM
I have no clue whether the warning is germaine or not, it just seemed like an obvious hint you weren't mentioning; so wanted to be sure it had been investigated.
a

average-sugar-68948

12/21/2022, 3:17 PM
I tried passing 
--python-version 3.9
to mypy, didn't help
but it still seems like mypy is potentially not running in the right environment, if it thinks it's python 3.7 by default?
e

enough-analyst-54434

12/21/2022, 3:25 PM
If you run with 
--keep-sandboxes=on_failure
there will be a line printed near the mypy failure that reveals a temporary directory you can examine. That will contain a `__run.sh`script that can be used to emulate and inspect what happened. See: https://www.pantsbuild.org/docs/rules-api-tips#debugging-look-inside-the-chroot That may be the best path forward in your debugging quest.
The more relevant link is probably https://www.pantsbuild.org/docs/troubleshooting#debug-tip-inspect-the-sandbox-with---keep-sandboxes - you're not hacking on rules here, but the concept applies generally for rooting out mystery in Pants actions.
a

adventurous-journalist-72920

12/21/2022, 4:13 PM
I think I went through something similar. Ended up adding 
types-protobuf
in [mypy].extra_type_stubs in 
pants.toml
(along with one specific lockfile for mypy and one for the extras). I don’t have the error anymore.
Along with what John suggested, you can add 
--show-traceback
in args, so Mypy gives more details on errors
a

average-sugar-68948

12/21/2022, 4:52 PM
I'm not seeing anything different/interesting yet. I made a small repo for repro: https://github.com/jonas25007/pants-proto-test
does 
./pants check ::
fail?
(adding extra_type_stubs is basically equivalent to what I was trying)
e

enough-analyst-54434

12/21/2022, 4:55 PM
It is not!
There are complicated details behind the scenes.
a

average-sugar-68948

12/21/2022, 4:55 PM
ah ok, well I tried it anyway and same result
e

enough-analyst-54434

12/21/2022, 4:55 PM
Ok.
a

average-sugar-68948

12/21/2022, 4:58 PM
Let me know if you have a chance to confirm repro, want to rule out something specific to my system
e

enough-analyst-54434

12/21/2022, 5:01 PM
Is your system dockerable? Then you yourself can root that out! Will I need to install proto binaries to try your example repo?
a

average-sugar-68948

12/21/2022, 9:33 PM
that's true, I'll try in docker. doesn't pants manage the proto deps?
a

adventurous-journalist-72920

12/21/2022, 9:51 PM
I’ve been able to reproduce with your repo. 
./pants export-codegen
writes transpiled python classes into 
dist/codegen
, is there any way to change that? That would probably solve the problem, or at least, help a lot.
a

average-sugar-68948

12/22/2022, 2:47 PM
I realized that I was trying in docker as well through my CI (an image based on continuumio/miniconda3:latest). Not sure what 
dist/codegen
would have to do with it - check is finding the .pyi's (like it's supposed to?), the stubs just aren't getting installed
a

adventurous-journalist-72920

12/23/2022, 2:04 PM
@average-sugar-68948 I believe the issue comes from Mypy, that introduced a breaking change a few versions ago, and I thought it was reverted… Anyway, adding an arg in mypy config should help:
Copy code
[mypy]
args = ["--namespace-packages"]
(in pants.toml)
a

average-sugar-68948

12/23/2022, 3:20 PM
Yes!! this works for me, thanks so much for hunting it down
🙏 1
While I have you here and the test repo, I noticed that proto dependencies don't seem to resolve for the 
run
goal. For example, 
./pants run run.py
in that test repo (pull for update) results in an import error, even though 
dependencies --transitive
lists the proto target properly. The 
test
goal doesn't seem to have this problem (
./pants test tests/
). Am I doing something wrong?
a

adventurous-journalist-72920

12/23/2022, 8:01 PM
I guess there is an interference created by the 
__init__.py
file within 
proto
folder. As of right now, it’s unclear to me why test works and run does not, I would have expected none to work. I need to dig more in order to give you a satisfying answer, but you can just remove the 
__init__.py
file from proto and try again 🙂
a

average-sugar-68948

01/03/2023, 2:58 PM
sorry just catching up from holidays - yes thanks, removing 
__init__.py
does the trick
Powered by