https://pantsbuild.org/ logo
Join Slack
Powered by
Hi, I'm trying to fit in a use case using the pant...
# plugins
b
Hi, I'm trying to fit in a use case using the pants protobuf integration and would like some guidance 🧵
It is related to this thread, but I'll explain what I'm trying to achieve.
We use shared common protos in our org that are distributed in various forms depending on the authoring repos. For e.g proto files are included in a jar from a Java repo, and in a pypi package from a Python repo, or simply in a zip archive.
Protobuf source files in a Pants repo need to import these shared protos.
For e.g
Copy code
syntax = "proto3";

package sample.pacakge.v1;

import "com/shared/protos/v1/options.proto";  # lives in a archive

message SampleMessage {
  option (com.shared.protos.v1.kafka_message) = "sample-topic";
  string name = 1;
  string def = 2;
}
I can write a rule to download the archive and extract proto files from it. But, how do I teach the 
protobuf_sources
target to include the downloaded protos files in its input digest
Is it even possible without making changes to the 
generate_python_from_protobuf
rule?
I've naively tried using the 
shell_command
target to download shared protos and make it a dependency of 
protobuf_sources
. But that did not work. Downloaded file are available in a 
python_tests
context, but not in a 
protobuf_sources
one
I've also tried writing a custom target with a codegen rule that downloads the shared protos. For e.g
Copy code
protobuf_remote_sources(
    name="deps",
    remote_packages=["shared-protos", "other-shared-protos"]
)

protobuf_sources(
    sources=["**/*.proto"],
    dependencies=[":deps"]
)
Copy code
class ProtoRemoteSourcesField(MultipleSourcesField):
    alias = "_sources"
    use_source_roots = False
    expected_num_files = 0


class GenerateProtobufFilesRequest(GenerateSourcesRequest):
    input = ProtoRemoteSourcesField
    output = ProtobufSourceField


@rule(desc="Downloading proto files from PyPi packages")
async def get_protobuf_remote_dependencies(
    request: GenerateProtobufFilesRequest,
) -> GeneratedSources:
But it looks like what I want to do will require changes to the generate_python_from_protobuf rule.
Any help/pointers is appreciated
👋 bump on this one 🙂 Just want to understand if what I'm trying to do is possible without forking the existing plugin
g
I think your approach seems sound and would likely work for directly compiling the protobufs. For using them as deps you'd likely need to ask the upstream rule to hydrate generated sources, in the rule you linked.
b
yeah. I was looking to see if I can do that without making changes to the existing codegen rule.
Open in Slack
PreviousNext
Powered by