Hi, so I was finally able to get pants running inside a RHEL based Docker container and after adding bare minimum code, I did

./pants tailor

and got following output:

(pants) [root@9a5752857756 pants-training]# ./pants tailor

Created src/client/BUILD:

- Add python_sources target client

- Add pex_binary target greeter_client

Created src/proto/BUILD:

- Add protobuf_sources target proto

Created src/server/BUILD:

- Add pex_binary target greeter_server

- Add python_requirements target reqs

- Add python_sources target server

(pants) [root@9a5752857756 pants-training]#

After that I did

./pants package src/server/greeter_server.py

and encountered following error:

(pants) [root@9a5752857756 pants-training]# ./pants package src/server/greeter_server.py

05:54:49.45 [ERROR] 1 Exception encountered:

IndexError: string index out of range

(pants) [root@9a5752857756 pants-training]#

Could someone please help? My goal is to generate the stubs from the proto files and publish them as python wheels so that I can pip install them and use them in client and server code. Following is how my repo looks:

(pants) [root@9a5752857756 pants-training]# /tree.sh
Tree of: /root/pants-training
.
|__pants
|__src
| |__proto
| | |__helloworld.proto
| | |__BUILD
| |__server
| | |__requirements.txt
| | |__greeter_server.py
| | |__BUILD
| |__client
| | |__greeter_client.py
| | |__BUILD
|__.pants.d
| |__exceptions.log
| |__exceptions.471.log
| |__run-tracker
| | |__pants_run_2022_08_19_05_55_17_757_5184323f0cce42809255925d6987ccbe
| | | |__logs
| | |__pants_run_2022_08_19_05_55_11_114_e3698008d5454c2aa6d584bf5680eaed
| | | |__logs
| | |__pants_run_2022_08_19_05_53_24_325_48f808a97d1c4c8aa6faad9261780e60
| | | |__logs
| | |__pants_run_2022_08_19_05_48_46_160_81dfa93f029b44f0bc191a8560c255c6
| | | |__logs
| | |__pants_run_2022_08_19_05_54_45_113_dee6bff2c02a414c94a754c60781580c
| | | |__logs
| | |__pants_run_2022_08_19_05_54_08_319_a3964cb6faa049c7917933ccfe3ede11
| | | |__logs
| |__pants.log
|__pants.toml
|__.pids
| |__855f83c10083
| | |__pantsd
| | | |__socket
| | | |__pid
| | | |__process_name
| | | |__fingerprint
| |__.lock.pantsd

Disclaimer: I have never used protobuf from pants. But if you want to create a python you need to define a

python_distribution

inside the BUILD file and then run `./pants package src/server`: https://www.pantsbuild.org/docs/python-distributions#the-python_distribution-target

Thanks but I want to create wheels from the stubs generated from proto file. How do I generate them using pants? Once those stubs are generated which will be .py files, I can define the

python_distribution

target for them and then package them as a wheel.

yes that’s what

python_distribution

does. In

dependencies

you need to add the dependency on underlying source code. In this case you can use the

python_sources

target called

server

which I guessed from this

Created src/server/BUILD:
 - Add pex_binary target greeter_server
 - Add python_requirements target reqs
 - Add python_sources target server

So

dependencies=[":server"]

So I need to make this change in

src/server/BUILD

?

yes if you want to package the

server

Ok, but I don't want to package the server. I want to package the code generated from proto file

src/proto/helloworld.proto

. The code isn't present here in the repo because it is not generated yet. I am trying to figure out how to generate that using pants and that's where I need help.

So it seems as long as you use the correct import statement, pants will auto generate it. As shown in this code https://github.com/pantsbuild/example-codegen/blob/main/src/python/protobuf_examples/simple_example_test.py

Ok, so does it mean pants will auto generate at run time based on import statement?

So if you package your server, it will run the auto generation at that time not during run time

Ok, let me shell out some more details of what I want to do: 1. Generate stubs from proto files (save them anywhere within the repo). There's an orange code block just below the blue code block you mentioned which says that the generated code will be in the same directory as proto files. I am not concerned about their location at this point. Trying to figure out the pants command for this purpose. 2. Package those stubs as a wheel so that both client and server can pip install them and use them. 3. Package client as another python wheel so that I can pip install the client to use it anywhere. 4. Package server as a Docker image and deploy it on a remote server. Once the server is deployed, I can pip install the client anywhere and connect to the server. So now if the code is generated at the time of packaging, would it solve my purpose is what I am wondering?

Yes it should work. Just to make sure, you will host that built wheel somewhere so that later you can install it anywhere you want.

Yes, I will host the wheel. But let's talk about client wheel - when I would pip install it, it would try to also install all the dependencies...just like how a normal pip install works. But because the dependency wasn't saved anywhere apart from pants own temp directory and because it wasn't saved anywhere, it would not be published anywhere for the pip to resolve the dependency. Did this make sense?

It would package all the underlying dependencies. So it will include the generated python files from protobuf also. As long as pants figure out the underlying dependency (which happens here based on the import statement), it will package it. As I said you will have to add explicit

dependencies=[:server]

in your

python_distribution

Ok, I think I need to try this. One last thing - why do I need to add

dependencies=[:server]

in server's own BUILD file

src/server/BUILD

. Server is dependent on proto.

python distribution needs dependency on some source code explicitly. This is bit of gotcha (at least for me).

Hi @refined-addition-53644, Good Morning! I am trying to run

./pants package src/server:grpc-server

and get this error. Do you have any idea?

Ok, so does it mean pants will auto generate at run time based on import statement?

Correct!

Another useful thing for debugging is to add an explicit dependency (in

dependencies=

) from the target using the stubs to the

protobuf_sources

target owning the .proto files

If I understand the above statement correctly, then I must add the dependencies of both client and server in the BUILD file of proto (assuming you are familiar with the repo structure I posted yesterday).

If I understand the above statement correctly, then I must add the dependencies of both client and server in the BUILD file of proto (assuming you are familiar with the repo structure I posted yesterday).

Well, both the client and the server should be inferring the dependency on the proto, we just need to debug why they are not

Pants usually infers dependencies based on

import

statements. The one case where it cannot is a

python_distribution

must have an explicit dependency at least on its entry point (usually Pants can infer the entry point’s dependencies, and so on)

I have a question from this repo for import statements. The python code

simple_example_test.py

in

src/python/protobuf_examples

imports the stubs from

simple_example.v1.person_pb2

. But according to this BUILD file, the stubs should be generating in

src/python

.

Yes, src/python is a “source root” (see https://www.pantsbuild.org/docs/source-roots)

Ok, so the first thing that came to my mind was also the source roots and then I checked the source roots for the example-codegen repo which are the following:

.
src
src/protobuf
src/python
src/thrift

The source roots and everything made sense but then following comment and code line confused me:

# This will generate files under `src/python`, rather than `src/protobuf`, which is convenient
    # so that we do not need to add `__init__.py` files to `src/protobuf`. See
    # <https://www.pantsbuild.org/docs/protobuf-python#protobuf-and-source-roots>
    python_source_root="src/python",

If the code is being generated in src/python then why are we importing it from

simple_example.v1

?