Hey Team, looking to write an external tool to support twirp, specifically using the twirpy protoc plugin. One requirement to use the tool is to install it via

go

with

go install <http://github.com/verloop/twirpy/protoc-gen-twirpy@latest|github.com/verloop/twirpy/protoc-gen-twirpy@latest>

Is it possible to use

go

like this in pants via an external tool? Or would we need to install

go

as an external tool first and then we can install plugins with it?

See https://github.com/pantsbuild/pants/pull/13985 for an example of how to install a Go tool for use in Pants.

this is great example I'll give that a shot! This is going to be codegen for

Python

Cool. So you won't be affected by https://github.com/pantsbuild/pants/issues/14258 then.

does this also handle installing

go

for the plugin run or is it required that the host machine has

go

installed?

The Go backend requires having

go

already available. We would like to return to Pants managing that but have not done that work yet. (There had been performance issues with unpacking the Go SDK in every execution sandbox.)

okay got it that make sense not a problem to have it preinstalled.

Is the call to register the plugin field actually being invoked?

hmm actually it doesn't seem to be calling the register for the plugin (it was before). Digging into this

are you able to put the sources somewhere?

ahh sorry that was a perma link this is the most recent file: https://github.com/njgrisafi/pants-example/blob/twirp-example/pants-plugins/python_twirp/rules.py

the input on the request type is wrong:

input = ProtobufTwirpField

should be

input = ProtobufSourceField

I tried that but get the following error:

njgrisafi@Nicks-MacBook-Pro ~/workspace/pants-example (twirp-example)$ ./pants export-codegen ::
19:40:59.75 [INFO] Initializing scheduler...
19:41:00.44 [INFO] Scheduler initialized.
19:41:00.52 [ERROR] 1 Exception encountered:

  AmbiguousCodegenImplementationsException: Multiple of the registered code generators can generate PythonSourceField from ProtobufSourceField. It is ambiguous which implementation to use.

Possible implementations:

  * GeneratePythonFromProtobufRequest
  * GeneratePythonFromTwirpRequest

you’ll need to disable the

pants.backend.codegen.protobuf.python

backend

njgrisafi@Nicks-MacBook-Pro ~/workspace/pants-example (twirp-example)$ ./pants export-codegen ::
19:43:50.95 [INFO] Initializing scheduler...
19:43:51.22 [INFO] Scheduler initialized.
19:43:51.28 [ERROR] 1 Exception encountered:

  MappingError: Failed to parse ./app/BUILD:
Name 'protobuf_sources' is not defined.

If you expect to see more symbols activated in the below list, refer to <https://www.pantsbuild.org/v2.9/docs/enabling-backends> for all available backends to activate.

All registered symbols: ['_python_requirements_file', 'archive', 'docker_image', 'file', 'files', 'pants_requirement', 'pants_requirements', 'pex_binary', 'pipenv_requirements', 'poetry_requirements', 'python_artifact', 'python_distribution', 'python_requirement', 'python_requirements', 'python_source', 'python_sources', 'python_test', 'python_test_utils', 'python_tests', 'relocated_files', 'resource', 'resources', 'setup_py', 'target']

Maybe I just need to register the rule for target type, think I know how to fix this

you should just register both protobuf target types in your twirp plugin in the

target_types

method

yeah Ideally I'd like to have the native python codegen + this additional codegen to generate

twirpy

originally I was thinking that this can probably be added to the existing plugin upstream

see https://github.com/pantsbuild/pants/blob/528f4b1911eed473f4fd371732d0f3c74561dcb2/src/python/pants/backend/codegen/protobuf/python/register.py#L[…]9

yeah the only gotcha with

twirpy

is you have to have that go plugin installed for it I believe

idea for better API: modify the existing Python protobuf backend to expose and use a union for RPC codegen backends to be called by the Python protobuf backend at the right time. then refactor grpc to use that union. then twirp could also use that union.

yeah that would be great for the API! Yeah and I don't want to diverge too far from upstream changes just to add one more feature but will work for now. Let me give that a go

and to refine the better API idea: the exposed union could be to allow grpc and twirp to add additional protoc plugins (and CLI args) to the protoc invocation

looks like this rule is outputting an error: https://github.com/njgrisafi/pants-example/blob/0bbaea7cfb8a39af8839829f85231bf91b49b763/pants-plugins/custom_python_protobuf/rules.py#L71-L93

njgrisafi@Nicks-MacBook-Pro ~/workspace/pants-example (twirp-example)$ ./pants export-codegen ::
20:11:23.01 [INFO] Initializing scheduler...
20:11:23.28 [INFO] Scheduler initialized.
20:11:23.80 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Build Go twirpy protobuf plugin for `protoc`.' failed with exit code 1.
stdout:

stderr:
go install: <http://github.com/verloop/twirpy/protoc-gen-twirpy@latest|github.com/verloop/twirpy/protoc-gen-twirpy@latest>: module lookup disabled by GOPROXY=off



Use `--no-process-cleanup` to preserve process chroots for inspection.