Hello how do I generate protobuf s python codes into the spe Pants #general

Hello, how do I generate protobuf’s python codes i...

freezing-quill-50372

03/09/2022, 1:52 AM

Hello, how do I generate protobuf’s python codes into the specific python root directory ? I already specified

python_source_root

, but the codes are generated in the original directory (detailed settings are following in thread)

freezing-quill-50372

03/09/2022, 1:54 AM

Copy code

├── apis
│   ├── README.md
│   └── protos
│       ├── BUILD
│       ├── README.md
│       └── some_proto.proto
├── python
│   ├── README.md
│   ├── pkg
│   │   ├── README.md
│   │   ├── apis
│   │   │   ├── BUILD
│   │   │   ├── README.md
│   │   │   ├── common_namespace
│   │   │   │   ├── README.md
│   │   │   │   └── apis
│   │   │   │       ├── __init__.py
│   │   │   │       ├── exceptions.py
│   │   │   │       └── protos
│   │   │   │           └── __init__.py <- I would like to generate the codes into this directory
│   │   │   ├── poetry.lock
│   │   │   ├── pyproject.toml
│   │   │   └── tests
│   │   │       └── __init__.py
...
│   ├── projects
│   │   ├── README.md
│   │   ├── some_project
│   │   │   ├── BUILD
│   │   │   ├── README.md
│   │   │   ├── poetry.lock
│   │   │   ├── pyproject.toml
│   │   │   ├── some_project
│   │   │   │   ├── __init__.py
│   │   │   │   ├── interface
│   │   │   │   │   ├── __init__.py
│   │   │   │   │   ├── cmd
│   │   │   │   │   │   └── __init__.py
│   │   │   │   │   └── grpc
│   │   │   │   │       ├── __init__.py
│   │   │   │   │       ├── main.py
│   │   │   │   │       └── servicier.py
...

freezing-quill-50372

03/09/2022, 1:54 AM

BUILD

in the

apis/protos

is following.

Copy code

protobuf_sources(
    name="protos",
    grpc=True,
    python_source_root="python/pkg/apis/common_namespace"
)

freezing-quill-50372

03/09/2022, 2:00 AM

some_project

’s

BUILD

refers the protobuf target as following.

Copy code

python_sources(
    name="pkg",
    sources=["some_project/**/*.py"],
    dependencies=[
        "apis/protos:protos",
        "python/pkg/utils:pkg",
        "python/pkg/clients:pkg",
        "python/pkg/apis:pkg",
        ":pydantic",
        ":grpcio",
        ":protobuf",
    ],
)

hundreds-father-404

03/09/2022, 2:03 AM

Hello! can you please run

./pants roots

and share it?

hundreds-father-404

03/09/2022, 2:03 AM

Also are you running

./pants export-codegen

to see where it's being generated into?

freezing-quill-50372

03/09/2022, 2:04 AM

Thanks! I’ll try them

freezing-quill-50372

03/09/2022, 2:05 AM

./pants root

result is here,

Copy code

python/pkg/apis
python/projects/some_project

freezing-quill-50372

03/09/2022, 2:07 AM

./pants export-codegen

’s result is here. It is odd, protobuf python codes themselves are generated when I packaged the

some_project

…

Copy code

02:06:14.36 [WARN] No codegen files/targets matched. All codegen target types: docker_image, protobuf_source

hundreds-father-404

03/09/2022, 2:10 AM

Oh to clarify,

./pants export-codegen ::

. You have to tell Pants what to run on

hundreds-father-404

03/09/2022, 2:10 AM

https://pantsbuild.slack.com/archives/C046T6T9U/p1646791558241399?thread_ts=1646790753.551439&cid=C046T6T9U Okay great. So,

python_source_root

needs to be set to one of those specifically. Without the extra folders at the end

freezing-quill-50372

03/09/2022, 2:14 AM

Great, thank you so much. I’ll try out

python/pkg/apis

. https://pantsbuild.slack.com/archives/C046T6T9U/p1646791840206159?thread_ts=1646790753.551439&cid=C046T6T9U

🤞 1

freezing-quill-50372

03/09/2022, 2:17 AM

By the way,

./pants export-codegen ::

said following. The following results seem to lack the namespace

some_namespace

Copy code

└── python
    ├── pkg
    │   └── apis
    │       └── apis
    │           └── protos
    │               ├── some_proto_pb2.py
    │               └── some_proto_pb2_grpc.py

freezing-quill-50372

03/09/2022, 2:20 AM

Then, Pants ignore implicit namespaces ?

hundreds-father-404

03/09/2022, 9:47 PM

That's right. Protobuf files are generated using their file path, plus some source root manipulation. Do you have with

./pants roots

an entry like

btw? If not, I'm surprised Pants is not erroring. Assuming that you have the source root

./pants roots

, then here is what's happening to get that path. 1. You have the file

api/protos/some_protos.proto

- it matches the source root

, which means there is no prefix to strip. So you stick with

api/protos/some_protos.proto

, which then generates the file

api/protos/some_protos_pb2.py

2. Because you are setting the field

python_source_root

, that gets added by Pants as a prefix. So you end up with

python/pkg/apis/api/protos/some_protos.proto

If you want to generate into a different directory, then you need to locate the

.proto

file in a different location like

common_namespace/apis/protos/some_proto.proto

Does that make sense?

Then, Pants ignore implicit namespaces ?

What do you mean? How are you doing this currently without Pants?

freezing-quill-50372

03/09/2022, 10:32 PM

Thanks for your detailed explanation, I got the process of proto codes generation.

What do you mean? How are you doing this currently without Pants?

Never mind, actually it seems to be my misunderstand. I’m building project structure from scratch for a new product and struggling how to confirm it.

❤️ 1

hundreds-father-404

03/09/2022, 10:58 PM

Ah okay, cool! Btw have you seen https://github.com/pantsbuild/example-codegen?

Open in Slack

Previous Next