https://pantsbuild.org/ logo
s

salmon-planet-80683

08/19/2021, 8:22 PM
Hi pants'ers! Had a quick question about the interplay between the protobuf backend and the mypy plugin. I have a
protobuf_library()
, and in my pants.toml I have
mypy_plugin = true
set for the protobuf backend. When I examine the chroot for
pants typecheck ::
(thanks to
--no-process-execution-local-cleanup
) I see that: ā€¢ my protobuf_library has accurately generated
.py
and
.pyi
files. ā€¢ Importantly, I do not see any `__init__.py`s in the protobuf-codegen'd directory. When I run
pants typecheck
, Mypy complains with: "Cannot find implementation or library stub for module named 'your_thing_that_you_imported_pb2'" Interestingly, at runtime (i.e. during a test) this import is resolved correctly. So, at this point, my suspicion is that the codegen is correct and mypy is just not able to find the
.pyi
files. I was browsing through mypy documentation and I came across a seemingly perfect flag:
--namespace-packages
- that accounts for packages lacking init.py. Sure enough, running it with this flag makes it all work swimmingly.
h

hundreds-father-404

08/19/2021, 8:27 PM
Hey Max, welcome! Sorry for the initial confusion - I have a TODO to better document the interplay of Protobuf and
__init__.py
files. I'm glad you were able to figure this out! FYI the other way you could have resolved this is adding empty
__init__.py
files to the locations where the files get generated to. Usually people don't want to add those to a
src/protos
folder, for example, hence the
python_source_root
option to allow you to have a file like
src/protos/project/models.proto
generate into
src/py/project/models_pb2.py
. Then you would have
src/py/project/__init__.py
and it should work. Does that make sense? Using namespace packages is also totally valid. Although I don't think we want to be activating it by default: we generally try to keep the magic of setting MyPy options to a minimum because it can result in confusion
šŸ™Œ 1
s

salmon-planet-80683

08/19/2021, 8:32 PM
Yep, that makes sense. I definitely noticed the docs have a lot of warnings of adding init.py - and sure enough, we have stuff in
src/protos
and are not using
python_source_root
.
fwiw the only repo I could find that tries to demonstrate this is https://github.com/danieljanes/pants-python-protobuf (you're a contributor!) the proto BUILD says the generated source root will be
src/py
- so I was a bit surprised that
src/py
doesn't, itself, have an init https://github.com/danieljanes/pants-python-protobuf/tree/master/src/py
h

hundreds-father-404

08/19/2021, 8:34 PM
Ohh I guess I did finish that TODO and never checked it off my list šŸ™ˆ https://www.pantsbuild.org/docs/protobuf#step-4-use-the-protobuf_library-in-your-python-code Any feedback on what was confusing there or could be improved? It sounds like a worked out example could help, e.g. linking to Daniel's repo
āœ… 1
s

salmon-planet-80683

08/19/2021, 8:34 PM
(slash, it'd be super cool and awesome if you guys could update this repo a bit [it's currently on some weird beta version of pants] and perhaps link it from the "Protobuf and gRPC" page šŸ™‚ )
yeah definitely a concrete example would help, with notes about "oh this is why this initpy is here"
šŸ‘ 1
Oh, one other question was: Let's say you have src/protos/BUILD
Copy code
protobuf_library(
  name="protos",
  python_source_root="src/python/protos-python"
)
if i wanted an explicit
dependencies=
declaration, would I specify
src/protos:protos
or
src/python/protos-python
?
p.s. as a former bazel user, love this product a hell of a lot more, keep it up šŸ™‚
h

hundreds-father-404

08/19/2021, 8:49 PM
You would depend on the original `protobuf_library`'s "address", so
src/protos:protos
. Or, use dependency inference which will infer a dep on
my_module_pb2
šŸ™‚
Glad to hear you're enjoying Pants! We've been putting a lot of focus on making it more ergonomic and joyful to get the benefits of build tools w/o all the boilerplate We still have more to go towards that vision, like making BUILD files optional most the time...but I appreciate the feedback!