brash-baker-9119002/03/2021, 6:26 PM
target (https://www.pantsbuild.org/docs/protobuf#protobuf-and-source-roots). The documentation suggests that this will put the generated code in the specified source root; however, in practice it looks like it just creates that directory structure under
Additionally, dependency resolution for code that's using the generated
modules doesn't appear to be working (as judged by the output of
Another wrinkle is that we'd like to create a Python distribution containing just the generated protobuf code, but I'm not even sure how to express that in Pants notation... particularly with the need to have
./pants dependencies path/to/the/file/that/uses/protobufs.py
files created as well.
There seem to be a lot of moving parts here, and I'm not sure if there's a bug or (more likely) some incomplete understanding on my part, possibly coupled with some inconsistent file layout choices in this repository.
That's a lot of information, and I can provide more as needed... I just wanted to lay out my perception of the high-level issues as a way to hopefully start a discussion. Thanks in advance for any advice folks may have.
happy-kitchen-8948202/03/2021, 6:34 PM
brash-baker-9119002/03/2021, 6:35 PM
hundreds-father-40402/03/2021, 6:36 PM
brash-baker-9119002/03/2021, 6:38 PM
hundreds-father-40402/03/2021, 6:39 PM
however, in practice it looks like it just creates that directory structure under dist/codegen.You're running
to get this, right? That is expected. That goal is purely for you to be able to view what files Pants is generating and to consume it externally, e.g. for IDE autocompletion.
You do not need to ever manually run that
goal for codegen to work. Pants runs things in chroots. It will include the generated files in those chroots for you, and it should always be up-to-date.
will show you what those final paths are, but with a
brash-baker-9119002/03/2021, 6:41 PM
, and I set that as a root in
and as a
, I should expect the generated code to ultimately reside in
and be dependency-resolved accordingly? That is, if I've got that as a directory containing just an empty
file, would the generated code then get recognized as part of a proper Python package?
hundreds-father-40402/03/2021, 6:50 PM
where it fails.
2. For that Python file's owning
target, add to the
field the path to that Protobuf file's directory, e.g.
. (This assumes that you have one BUILD file per directory, with empty
When running something like
, you can use
so that you can preserver the tmpdir and look to see that the paths are showing up how you'd expect
No need to get all imports of Protobuf working - we only need to get at least one working, so that we can isolate the problem.
I should expect the generated code to ultimately reside in foo/bar/bazYes. If you have
, and you set the
, then Pants should be creating the file
if I've got that as a directory containing just an empty init.py fileHmm, it's very plausible issues with
are at play. Let's start with confirming the
file shows up properly, then we can check if the
files are showing up
brash-baker-9119002/03/2021, 6:52 PM
happy-kitchen-8948202/03/2021, 7:24 PM
brash-baker-9119002/03/2021, 7:40 PM
file, and the protobuf code generation drops the new files off in that directory, and I can use them. The automatic dependency resolution doesn't appear to work, though, and I have to manually add it. Additionally, I can't seem to build a distribution for code that uses the generated code (let alone a distribution for just the generated code) because Pants says that no
owns the proto file in question.
hundreds-father-40402/03/2021, 7:49 PM
I suppose I can deal with that 🙂I'm glad there's a workaround, but that's not very satisfying and we'd love to figure this out. A major goal for Pants is that you do not need to reorganize your codebase (at least much) to get Pants working. And we have more work to do if that is not the case. I don't think
will work in the context of Protobuf; that won't trigger codegen, and it only works on files that you essentially want to have unmodified, outside of their path. Like JSON files or PNG files
brash-baker-9119002/03/2021, 7:54 PM
to direct where to put the entire package. That seems kind of niche, and a byproduct of us doing things incorrectly with protobufs. As such, I don't really view it as a problem with Pants. We need to clean up our protobufs for other reasons anyway; this merely gives us another reason to do it sooner 😅
hundreds-father-40402/04/2021, 4:27 PM
our protobuf file structure on disk didn't really match the protobuf package structureWhat does this look like? My experience with Protobuf is limited, and I've only seen it where the
clause == the directory path (minus source roots)
brash-baker-9119002/04/2021, 4:49 PM
hundreds-father-40402/04/2021, 4:52 PM