https://pantsbuild.org/ logo
#general
Title
# general
c

clean-window-81064

02/03/2023, 9:20 PM
Hey All, I am getting a weird error while trying to package a script containing some proto code as a wheel file. Some of the proto files are being packaged inside the wheels just fine, whereas some of them are being skipped even after being explicitly specified in the BUILD file. Any ideas/thoughts what might be happening here ??
c

curved-television-6568

02/03/2023, 9:44 PM
Are all your proto files accounted for when running
pants dependencies --transitive path/to/your/distribution:target
?
h

happy-kitchen-89482

02/03/2023, 9:45 PM
when you say proto files, are the
.proto
files expected to be in the wheel, or the
_pb.py
files generated from them?
c

clean-window-81064

02/04/2023, 7:00 AM
Andreas - Yes, I can see all the files being accounted for. Benjy - I meant the _pb.py files being generated from the proto.
h

happy-kitchen-89482

02/04/2023, 3:42 PM
Are the missing _pb.py files generated to a path that is under a source root (https://www.pantsbuild.org/docs/source-roots)?
Do you see them when you run
./pants export-codegen
?
Can you think of any differences between the ones that are in the wheel and the ones that aren't?
And I assume you've checked (with
unzip -t
, say) that the files are actually not in the wheel file? rather than just not importing for some reason?
c

clean-window-81064

02/04/2023, 4:48 PM
Are the missing _pb.py files generated to a path that is under a source root
Yes
Do you see them when you run
./pants export-codegen
?
oh I wasn’t aware of this Thanks. I tried this & am running into some errors related to golang -
Copy code
Please specify either:
        • a "go_package" option in the .proto source file, or
        • a "M" argument on the command line.

See <https://developers.google.com/protocol-buffers/docs/reference/go-generated#package> for more information.

--go_out: protoc-gen-go: Plugin failed with status code 1.
I will try to figure a way around this and share result. In the meantime, if you have any suggestions pls let me know. (EDIT: Check below added notes)
Can you think of any differences between the ones that are in the wheel and the ones that aren’t?
No, not really. Both are defined in the BUILD file, and in some other
pex_binary
target they are being imported as expected. One thing is - the missing files (let’s call it depA.proto) is being imported in other parent proto file (let’s call is parent.proto) & the parent.proto is being imported as expected (but missing the dependency).
And I assume you’ve checked (with
unzip -t
, say) that the files are actually not in the wheel file?
Yes that’ right. I have unzipped the tar.gz and checked EDIT: The
codegen
was resolved and it was able to generate all the protos.
This was resolved - as per my limited understanding, the issues was the structure of the package. The
python_distribution
wasn’t defined at the root and thus it was failing to pull some files. Reason for not starting with
python_distribution
in the root itself was - we already have a
python_distribution
defined at the root & thus while adding another one was getting
AmbiguousOwnerError
, but seems like there is no other way here ? Exact error -
Copy code
AmbiguousOwnerError: Found multiple sibling python_distribution targets that are the closest ancestor dependees of src/backend/abc/libs/service/quitter.py:quitter and are therefore candidates to own it: src/backend/abc:rl, src/backend/abc:data_processing. Only a single such owner is allowed, to avoid ambiguity. See <https://www.pantsbuild.org/v2.14/docs/python-distributions> for how python_sources targets are mapped to distributions. See <https://www.pantsbuild.org/v2.14/docs/python-distributions>.
c

curved-television-6568

02/06/2023, 1:50 PM
if you want to keep the project structure, you may exclude files that should go into the other distrubution by adding
!!src/backend/abc/libs/service/quitter.py
to the dependencies of conflicting distribution. https://www.pantsbuild.org/docs/targets#dependencies-field
c

clean-window-81064

02/07/2023, 10:09 AM
Thanks Andreas, I tried that but there are some common files that needs to be shared across both the distributions so I would probably need to find some other way here
✔️ 1
h

happy-kitchen-89482

02/07/2023, 5:27 PM
Ah yes, when Pants generates setup.py etc. for you from a
python_distribution
, it has to have some way of assigning each python module to a wheel it builds. The algorithm it uses for this is described here: https://www.pantsbuild.org/docs/python-distributions#mapping-source-files-to-distributions
Your layout was in violation of the assumptions of that algorithm...
You can always write your own setup.py/setup.cfg, but that is of course more laborious
c

clean-window-81064

02/09/2023, 6:27 PM
Yep, I am currently going that route only. 😄
7 Views