Hmm, 2.11.0rc1 isn't letting me use the protobuf codegen for Go. I get

No module named 'pants.backend.experimental.codegen.protobuf.go'

. Is it because it's missing from here perhaps?

yeah that would be it. I must not have added it to the init/BUILD in the PR that purported to provide that backend. not the first time I left something out of init/BUILD. 🤦

i am still finding it necessary to generate the *.pb.go files in advance otherwise pants will fail https://github.com/xyzst/go-mono-repo-with-pants-sample What am I missing?

143056.75 [WARN] Unmatched globs from pkg/model:model0's

sources

field: [“pkg/model/*.go”, “pkg/model/*.s”]

Did you remove the corresponding

go_package

?

PANTS_SHA=529979dc04fca1f5aea1ea0cb23ab514f11abe32 ./pants package ::
17:27:54.39 [INFO] Initializing scheduler...
17:27:54.45 [INFO] Scheduler initialized.
17:27:56.67 [ERROR] Completed: Compile with Go - main failed (exit code 1).
./cmd/client/main.go:9:5: could not import <http://github.com/xyzst/go-mono-repo-with-pants-sample/pkg/model|github.com/xyzst/go-mono-repo-with-pants-sample/pkg/model> (open : no such file or directory)

17:27:56.67 [ERROR] Completed: Compile with Go - main failed (exit code 1).
./cmd/server/main.go:10:5: could not import <http://github.com/xyzst/go-mono-repo-with-pants-sample/pkg/model|github.com/xyzst/go-mono-repo-with-pants-sample/pkg/model> (open : no such file or directory)

17:27:56.67 [ERROR] 1 Exception encountered:

  Exception: Failed to compile main:
./cmd/client/main.go:9:5: could not import <http://github.com/xyzst/go-mono-repo-with-pants-sample/pkg/model|github.com/xyzst/go-mono-repo-with-pants-sample/pkg/model> (open : no such file or directory)

i removed it, still the same

can you share your BUILD files for the cmd/client and pkg/model directories? and also what the files are (if you can’t share names, then just number *.go and other file extensions)

@fast-nail-55400: https://github.com/xyzst/go-mono-repo-with-pants-sample

option go_package = "pkg/model";

— make that be the full import path, so

option go_package = "<http://github.com/xyzst/go-mono-repo-with-pants-sample/pkg/model|github.com/xyzst/go-mono-repo-with-pants-sample/pkg/model>";

thanks for the help! it worked

do you know whether

option go_package = "pkg/model"

should have worked as written? i.e., does

protoc

prepend the import path from the go.mod at all?

yes it works as written. if you run

protoc --go_out=./ --go_opt=paths=source_relative \
    --go-grpc_out=./ --go-grpc_opt=paths=source_relative \
    ./pkg/model/hello_world.proto

Finally giving this feature a new try in a repo with Go and Python code. When Go code references a generated protobuf in the wrong location from what the

option go_package

specifies I get the following error (as expected) when running `./pants run path/to/:bin`:

Exception: Failed to compile main:
./path/to/file.go:3:16: could not import <http://example.com/invalid/path/to/generated/proto|example.com/invalid/path/to/generated/proto> (open : no such file or directory)

When I fix this path however to what's in

option go_package

, I get this interesting error when running `./pants run path/to/:bin`:

KeyError: 'The target `python_requirement(address="vendor#grpcio", dependencies=(\'vendor/requirements.txt\',), description=None, modules=(\'grpc\',), requirements=(<pants.backend.python.pip_requirement.PipRequirement object at 0x105877d30>,), resolve=None, tags=None, type_stub_modules=None)` does not have a field `ProtobufSourceField`. Before calling `my_tgt[ProtobufSourceField]`, call `my_tgt.has_field(ProtobufSourceField)` to filter out any irrelevant Targets or call `my_tgt.get(ProtobufSourceField)` to use the default Field value.'

./pants export-codegen

does work, so seems like the Go dependency inference is the cause of it rather than the code generation itself. Any way to further debug this?

what is

option go_package

set to? full import path?

option go_package

is full import, yes. It's the exact path the .proto-files are in, but with the repository's URL prepended.

Engine traceback:
  in select
  in pants.core.goals.run.run
  in pants.backend.go.goals.run_binary.create_go_binary_run_request (golang/cmd/planner:bin)
  in pants.backend.go.goals.package_binary.package_go_binary (golang/cmd/planner:bin)
  in pants.backend.go.util_rules.build_pkg.required_built_go_package (golang/cmd/planner:planner)
  in pants.backend.go.util_rules.build_pkg.build_go_package (golang/cmd/planner:planner)
  in pants.backend.go.util_rules.build_pkg_target.required_build_go_package_request (golang/cmd/planner:planner)
  in pants.backend.go.util_rules.build_pkg_target.setup_build_go_package_target_request (golang/cmd/planner:planner)
  in pants.backend.go.util_rules.build_pkg_target.setup_build_go_package_target_request (path/to/some/protobuf/file.proto)
  in pants.backend.codegen.protobuf.go.rules.setup_build_go_package_request_for_protobuf
  in pants.backend.codegen.protobuf.go.rules.setup_full_package_build_request
Traceback (most recent call last):
  File "/Users/jyggen/.cache/pants/setup/bootstrap-Darwin-arm64/2.11.0rc2_py39/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 705, in native_engine_generator_send
    res = func.send(arg)
  File "/Users/jyggen/.cache/pants/setup/bootstrap-Darwin-arm64/2.11.0rc2_py39/lib/python3.9/site-packages/pants/backend/codegen/protobuf/go/rules.py", line 198, in setup_full_package_build_request
    SourceFilesRequest(
  File "/Users/jyggen/.cache/pants/setup/bootstrap-Darwin-arm64/2.11.0rc2_py39/lib/python3.9/site-packages/pants/util/meta.py", line 130, in new_init
    prev_init(self, *args, **kwargs)
  File "/Users/jyggen/.cache/pants/setup/bootstrap-Darwin-arm64/2.11.0rc2_py39/lib/python3.9/site-packages/pants/core/util_rules/source_files.py", line 42, in __init__
    self.sources_fields = tuple(sources_fields)
  File "/Users/jyggen/.cache/pants/setup/bootstrap-Darwin-arm64/2.11.0rc2_py39/lib/python3.9/site-packages/pants/backend/codegen/protobuf/go/rules.py", line 199, in <genexpr>
    sources_fields=(tgt[ProtobufSourceField] for tgt in transitive_targets.closure),
  File "/Users/jyggen/.cache/pants/setup/bootstrap-Darwin-arm64/2.11.0rc2_py39/lib/python3.9/site-packages/pants/engine/target.py", line 483, in __getitem__
    raise KeyError(
KeyError: 'The target `python_requirement(address="vendor#grpcio", dependencies=(\'vendor/requirements.txt\',), description=None, modules=(\'grpc\',), requirements=(<pants.backend.python.pip_requirement.PipRequirement object at 0x10f3b3100>,), resolve=None, tags=None, type_stub_modules=None)` does not have a field `ProtobufSourceField`. Before calling `my_tgt[ProtobufSourceField]`, call `my_tgt.has_field(ProtobufSourceField)` to filter out any irrelevant Targets or call `my_tgt.get(ProtobufSourceField)` to use the default Field value.'

https://github.com/pantsbuild/pants/blob/bc4cd049878fd0ee07fe0a9c84faa5cde352e75d/src/python/pants/backend/codegen/protobuf/go/rules.py#L199 needs a

if tgt.has_Field(ProtobufSourceField)

on the comprehension

I do have the former at least. Let me give it a try in a little bit, about to head out 🙂

there are some examples of

pants_from_sources

in some of the Pants example repos, e.g. https://github.com/pantsbuild/example-codegen/blob/main/pants_from_sources

Yep, can confirm that fixes the issue. Thanks! 🙂

Cool. Good to know!