Hello 👋 I'm currently exploring Pants for our Python monorepo at work, and am running into some difficulties with the protobuf code generation part of things. I'm not sure if this is a bug, misconfiguration and misunderstanding on my part, or something else, so any help is appreciated. At a high level, we have some protobufs that we need to compile into Python code and into Rust code (compiling the Rust code is pretty straightforward and isn't part of what I'm looking for here; we're currently looking at Pants to help manage the Python side of our codebase.) Additionally, we'd like to package up the generated Python code into a distribution. Since the protobuf definitions are off on their own, and since we generate code for more than just Python, I don't want the generated code to be dropped into the same directory. I've looked at the documentation that suggests adding

python_source_root

to the

protobuf_library

target (https://www.pantsbuild.org/docs/protobuf#protobuf-and-source-roots). The documentation suggests that this will put the generated code in the specified source root; however, in practice it looks like it just creates that directory structure under

dist/codegen

. Additionally, dependency resolution for code that's using the generated

*_pb2

modules doesn't appear to be working (as judged by the output of

./pants dependencies path/to/the/file/that/uses/protobufs.py

Another wrinkle is that we'd like to create a Python distribution containing just the generated protobuf code, but I'm not even sure how to express that in Pants notation... particularly with the need to have

__init__.py

files created as well. There seem to be a lot of moving parts here, and I'm not sure if there's a bug or (more likely) some incomplete understanding on my part, possibly coupled with some inconsistent file layout choices in this repository. That's a lot of information, and I can provide more as needed... I just wanted to lay out my perception of the high-level issues as a way to hopefully start a discussion. Thanks in advance for any advice folks may have.

Hi Chris, welcome to Pants! I'll try and sort through the issue here, but will also let @hundreds-father-404 chime in if I get anything wrong.

@happy-kitchen-89482 Thanks very much 👍

Hey Chris! Welcome. First, to confirm, are you using 2.2 or newer? That's when we added dep inference for Protobuf

Thanks @hundreds-father-404 👋

however, in practice it looks like it just creates that directory structure under dist/codegen.

You're running

./pants export-codegen

to get this, right? That is expected. That goal is purely for you to be able to view what files Pants is generating and to consume it externally, e.g. for IDE autocompletion. You do not need to ever manually run that

export-codegen

goal for codegen to work. Pants runs things in chroots. It will include the generated files in those chroots for you, and it should always be up-to-date.

export-codegen

will show you what those final paths are, but with a

dist/codegen

prefix.

OK, then I think that bit should be OK.

Re dep inference, for now, try to manually add the deps and see if we can get the runtime working. I suspect that something with source roots could be off, and that getting the manual thing working will get dep inference working 1. Choose a Python file that uses Protobuf, and some simple thing you can run with it like

./pants test

where it fails. 2. For that Python file's owning

python_library

target, add to the

dependencies

field the path to that Protobuf file's directory, e.g.

src/protos/models

. (This assumes that you have one BUILD file per directory, with empty

python_library()

and

protobuf_library()

definitions) When running something like

./pants test

, you can use

--no-process-execution-cleanup-local-dirs

so that you can preserver the tmpdir and look to see that the paths are showing up how you'd expect No need to get all imports of Protobuf working - we only need to get at least one working, so that we can isolate the problem.

OK, I'll give that a shot and report back; thanks!

My work is done... 🙂

🤔 I got a test working after renaming a bunch of directories (our code layout is a bit muddled at the moment 😕) I have my directory containing just an

__init__.py

file, and the protobuf code generation drops the new files off in that directory, and I can use them. The automatic dependency resolution doesn't appear to work, though, and I have to manually add it. Additionally, I can't seem to build a distribution for code that uses the generated code (let alone a distribution for just the generated code) because Pants says that no

python_distribution

owns the proto file in question.

I suppose I can deal with that 🙂

I'm glad there's a workaround, but that's not very satisfying and we'd love to figure this out. A major goal for Pants is that you do not need to reorganize your codebase (at least much) to get Pants working. And we have more work to do if that is not the case. I don't think

relocated_files

will work in the context of Protobuf; that won't trigger codegen, and it only works on files that you essentially want to have unmodified, outside of their path. Like JSON files or PNG files

Sure... it may take a little while to get things cleaned up and sort out exactly which parts were contributing to the behavior, but once that's sorted out, I can come up with a reproduction.

Interestinggg! I didn't know that's a thing.

our protobuf file structure on disk didn't really match the protobuf package structure

What does this look like? My experience with Protobuf is limited, and I've only seen it where the

package

clause == the directory path (minus source roots)

That's basically what it should look like, and I think it's what most tooling in the ecosystem expects. I'm not quite sure how ours came to be how it currently is, but it had a package name that wasn't the same as the directory it was in, and we also wanted it to be nested within another directory. Kinda wacky.

Awesome, thanks for the update! Let us know if you're still having issues with things, or more general feedback / feature requests