https://pantsbuild.org/ logo
#development
Title
# development
n

narrow-vegetable-37489

04/01/2022, 11:15 AM
Hmm, 2.11.0rc1 isn't letting me use the protobuf codegen for Go. I get
No module named 'pants.backend.experimental.codegen.protobuf.go'
. Is it because it's missing from here perhaps?
f

fast-nail-55400

04/01/2022, 11:56 AM
yeah that would be it. I must not have added it to the init/BUILD in the PR that purported to provide that backend. not the first time I left something out of init/BUILD. 🤦
👍 2
I’ll back-port to 2.11.x branch as well. Even without a 2.11.x release, you will be able to consume it via the
PANTS_SHA
mechanism in the
./pants
script.
h

hallowed-plastic-91664

04/02/2022, 7:28 PM
i am still finding it necessary to generate the *.pb.go files in advance otherwise pants will fail https://github.com/xyzst/go-mono-repo-with-pants-sample What am I missing?
after deleting the *.pb.go files locally, I get this:
Copy code
PANTS_SHA=529979dc04fca1f5aea1ea0cb23ab514f11abe32 ./pants package ::
14:30:56.75 [WARN] Unmatched globs from pkg/model:model0's `sources` field: ["pkg/model/*.go", "pkg/model/*.s"]

Do the file(s) exist? If so, check if the file(s) are in your `.gitignore` or the global `pants_ignore` option, which may result in Pants not being able to see the file(s) even though they exist on disk. Refer to <https://www.pantsbuild.org/v2.12/docs/troubleshooting#pants-cannot-find-a-file-in-your-project>.
14:30:56.75 [ERROR] Completed: Set up Go compilation request - <http://github.com/xyzst/go-mono-repo-with-pants-sample/pkg/model|github.com/xyzst/go-mono-repo-with-pants-sample/pkg/model> failed (exit code 1).
failed to read directory pkg/model: open pkg/model: no such file or directory

14:30:56.75 [ERROR] 1 Exception encountered:

  Exception: Failed to determine metadata to compile main:
failed to read directory pkg/model: open pkg/model: no such file or directory
f

fast-nail-55400

04/04/2022, 2:19 PM
143056.75 [WARN] Unmatched globs from pkg/model:model0's
sources
field: [“pkg/model/*.go”, “pkg/model/*.s”]
Did you remove the corresponding
go_package
?
h

hallowed-plastic-91664

04/04/2022, 10:28 PM
Copy code
PANTS_SHA=529979dc04fca1f5aea1ea0cb23ab514f11abe32 ./pants package ::
17:27:54.39 [INFO] Initializing scheduler...
17:27:54.45 [INFO] Scheduler initialized.
17:27:56.67 [ERROR] Completed: Compile with Go - main failed (exit code 1).
./cmd/client/main.go:9:5: could not import <http://github.com/xyzst/go-mono-repo-with-pants-sample/pkg/model|github.com/xyzst/go-mono-repo-with-pants-sample/pkg/model> (open : no such file or directory)

17:27:56.67 [ERROR] Completed: Compile with Go - main failed (exit code 1).
./cmd/server/main.go:10:5: could not import <http://github.com/xyzst/go-mono-repo-with-pants-sample/pkg/model|github.com/xyzst/go-mono-repo-with-pants-sample/pkg/model> (open : no such file or directory)

17:27:56.67 [ERROR] 1 Exception encountered:

  Exception: Failed to compile main:
./cmd/client/main.go:9:5: could not import <http://github.com/xyzst/go-mono-repo-with-pants-sample/pkg/model|github.com/xyzst/go-mono-repo-with-pants-sample/pkg/model> (open : no such file or directory)
i removed it, still the same
f

fast-nail-55400

04/04/2022, 11:33 PM
can you share your BUILD files for the cmd/client and pkg/model directories? and also what the files are (if you can’t share names, then just number *.go and other file extensions)
(The context being to try and produce a minimal reproduction of the issue.)
h

hallowed-plastic-91664

04/05/2022, 10:16 PM
if you clone that repo, then run the following command at the root proj dir:
Copy code
PANTS_SHA=529979dc04fca1f5aea1ea0cb23ab514f11abe32 ./pants package ::
It will produce that output
f

fast-nail-55400

04/06/2022, 12:34 AM
option go_package = "pkg/model";
— make that be the full import path, so
option go_package = "<http://github.com/xyzst/go-mono-repo-with-pants-sample/pkg/model|github.com/xyzst/go-mono-repo-with-pants-sample/pkg/model>";
gets further:
Copy code
22:02:05.20 [ERROR] Completed: Compile with Go - main failed (exit code 1).
./cmd/client/main.go:31:10: undefined: model.NewGreeterClient

22:02:05.20 [ERROR] Completed: Compile with Go - main failed (exit code 1).
./cmd/server/main.go:20:5: undefined: model.UnimplementedGreeterServer
./cmd/server/main.go:36:5: undefined: model.RegisterGreeterServer

22:02:05.20 [ERROR] 1 Exception encountered:

  Exception: Failed to compile main:
./cmd/client/main.go:31:10: undefined: model.NewGreeterClient
needs
grpc=True
on
protobuf_sources
Untitled.diff
🙌 1
h

hallowed-plastic-91664

04/06/2022, 3:30 PM
thanks for the help! it worked
f

fast-nail-55400

04/06/2022, 3:51 PM
do you know whether
option go_package = "pkg/model"
should have worked as written? i.e., does
protoc
prepend the import path from the go.mod at all?
(not that
protoc
knows anything about go.mod’s)
h

hallowed-plastic-91664

04/06/2022, 5:53 PM
yes it works as written. if you run
Copy code
protoc --go_out=./ --go_opt=paths=source_relative \
    --go-grpc_out=./ --go-grpc_opt=paths=source_relative \
    ./pkg/model/hello_world.proto
then the go build command:
go build cmd/client/main.go -o client
the go_package is more of a hint to protoc, but i do not think it is required
(ignore that, it’s required)
n

narrow-vegetable-37489

04/11/2022, 4:11 PM
Finally giving this feature a new try in a repo with Go and Python code. When Go code references a generated protobuf in the wrong location from what the
option go_package
specifies I get the following error (as expected) when running `./pants run path/to/:bin`:
Copy code
Exception: Failed to compile main:
./path/to/file.go:3:16: could not import <http://example.com/invalid/path/to/generated/proto|example.com/invalid/path/to/generated/proto> (open : no such file or directory)
When I fix this path however to what's in
option go_package
, I get this interesting error when running `./pants run path/to/:bin`:
Copy code
KeyError: 'The target `python_requirement(address="vendor#grpcio", dependencies=(\'vendor/requirements.txt\',), description=None, modules=(\'grpc\',), requirements=(<pants.backend.python.pip_requirement.PipRequirement object at 0x105877d30>,), resolve=None, tags=None, type_stub_modules=None)` does not have a field `ProtobufSourceField`. Before calling `my_tgt[ProtobufSourceField]`, call `my_tgt.has_field(ProtobufSourceField)` to filter out any irrelevant Targets or call `my_tgt.get(ProtobufSourceField)` to use the default Field value.'
./pants export-codegen
does work, so seems like the Go dependency inference is the cause of it rather than the code generation itself. Any way to further debug this?
f

fast-nail-55400

04/11/2022, 4:31 PM
what is
option go_package
set to? full import path?
also can you run with
--print-stacktrace
so we can see what code is triggered that KeyError
n

narrow-vegetable-37489

04/11/2022, 4:50 PM
option go_package
is full import, yes. It's the exact path the .proto-files are in, but with the repository's URL prepended.
Copy code
Engine traceback:
  in select
  in pants.core.goals.run.run
  in pants.backend.go.goals.run_binary.create_go_binary_run_request (golang/cmd/planner:bin)
  in pants.backend.go.goals.package_binary.package_go_binary (golang/cmd/planner:bin)
  in pants.backend.go.util_rules.build_pkg.required_built_go_package (golang/cmd/planner:planner)
  in pants.backend.go.util_rules.build_pkg.build_go_package (golang/cmd/planner:planner)
  in pants.backend.go.util_rules.build_pkg_target.required_build_go_package_request (golang/cmd/planner:planner)
  in pants.backend.go.util_rules.build_pkg_target.setup_build_go_package_target_request (golang/cmd/planner:planner)
  in pants.backend.go.util_rules.build_pkg_target.setup_build_go_package_target_request (path/to/some/protobuf/file.proto)
  in pants.backend.codegen.protobuf.go.rules.setup_build_go_package_request_for_protobuf
  in pants.backend.codegen.protobuf.go.rules.setup_full_package_build_request
Traceback (most recent call last):
  File "/Users/jyggen/.cache/pants/setup/bootstrap-Darwin-arm64/2.11.0rc2_py39/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 705, in native_engine_generator_send
    res = func.send(arg)
  File "/Users/jyggen/.cache/pants/setup/bootstrap-Darwin-arm64/2.11.0rc2_py39/lib/python3.9/site-packages/pants/backend/codegen/protobuf/go/rules.py", line 198, in setup_full_package_build_request
    SourceFilesRequest(
  File "/Users/jyggen/.cache/pants/setup/bootstrap-Darwin-arm64/2.11.0rc2_py39/lib/python3.9/site-packages/pants/util/meta.py", line 130, in new_init
    prev_init(self, *args, **kwargs)
  File "/Users/jyggen/.cache/pants/setup/bootstrap-Darwin-arm64/2.11.0rc2_py39/lib/python3.9/site-packages/pants/core/util_rules/source_files.py", line 42, in __init__
    self.sources_fields = tuple(sources_fields)
  File "/Users/jyggen/.cache/pants/setup/bootstrap-Darwin-arm64/2.11.0rc2_py39/lib/python3.9/site-packages/pants/backend/codegen/protobuf/go/rules.py", line 199, in <genexpr>
    sources_fields=(tgt[ProtobufSourceField] for tgt in transitive_targets.closure),
  File "/Users/jyggen/.cache/pants/setup/bootstrap-Darwin-arm64/2.11.0rc2_py39/lib/python3.9/site-packages/pants/engine/target.py", line 483, in __getitem__
    raise KeyError(
KeyError: 'The target `python_requirement(address="vendor#grpcio", dependencies=(\'vendor/requirements.txt\',), description=None, modules=(\'grpc\',), requirements=(<pants.backend.python.pip_requirement.PipRequirement object at 0x10f3b3100>,), resolve=None, tags=None, type_stub_modules=None)` does not have a field `ProtobufSourceField`. Before calling `my_tgt[ProtobufSourceField]`, call `my_tgt.has_field(ProtobufSourceField)` to filter out any irrelevant Targets or call `my_tgt.get(ProtobufSourceField)` to use the default Field value.'
f

fast-nail-55400

04/11/2022, 4:53 PM
do you have a pants checkout to the side of your repository (and a
./pants_from_sources
script)?
would be easy to test then
n

narrow-vegetable-37489

04/11/2022, 5:00 PM
I do have the former at least. Let me give it a try in a little bit, about to head out 🙂
👍 1
f

fast-nail-55400

04/11/2022, 5:13 PM
there are some examples of
pants_from_sources
in some of the Pants example repos, e.g. https://github.com/pantsbuild/example-codegen/blob/main/pants_from_sources
👍 1
n

narrow-vegetable-37489

04/12/2022, 12:30 AM
Yep, can confirm that fixes the issue. Thanks! 🙂
f

fast-nail-55400

04/12/2022, 12:40 AM
Cool. Good to know!