I’m trying to package a `python_distribution` that...
# general
l
I’m trying to package a
python_distribution
that depends on some
protobuf_sources
in a sibling directory. What do I need to specify in my dependencies field in order to satisfy the ownership constraint? Right now I have the following:
Copy code
python_sources(name="lib")

python_distribution(
    name="auth_service_client",
    dependencies=[
        ":lib",
        "src/proto/auth_service/service.proto:auth_service_interfaces",
    ],
    provides=python_artifact(
        name="auth_service_client",
        version="0.0.1",
    ),
)
This is in the subdirectory that has my python sources for the library. It depends on some codegen protocol buffers in a sibling directory with the following BUILD file:
Copy code
protobuf_sources(
    name="auth_service_interfaces",
    python_source_root="src/python",
    grpc=True,    
)
Pants is aware of the dependency between the
python_source
generated by `python_sources`:
Copy code
./pants dependencies src/python/auth_service_client/client.py
src/proto/auth_service/service.proto:auth_service_interfaces
src/python:reqs#grpcio
But when I try to package, I get the following error message:
Copy code
NoOwnerError: No python_distribution target found to own src/proto/auth_service/service.proto:auth_service_interfaces. Note that the owner must be in or above the owned target's directory, and must depend on it (directly or indirectly). See <https://www.pantsbuild.org/v2.14/docs/python-distributions> for how python_sources targets are mapped to distributions. See <https://www.pantsbuild.org/v2.14/docs/python-distributions>.
I read through the docs but I’m not understanding how to map the
protobuf_sources
to a
python_sources
target that my distribution can depend on. What am I missing here?
πŸ™Œ 1
e
"Note that the owner must be in or above the owned target's directory" - that's the missing bit. You have sibling directories. There is a bit of a long explanation about why this constraint exists, but that's somewhat immaterial and I'll stop there for now.
So you'd need to move the
python_distribution
target to a BUILD file that satisfies that constraint.
l
Got it. So if my tree looks like this:
Copy code
.
β”œβ”€β”€ pants
β”œβ”€β”€ pants.toml
└── src
    β”œβ”€β”€ go
    β”‚   β”œβ”€β”€ BUILD
    β”‚   β”œβ”€β”€ cmd
    β”‚   β”‚   └── auth-server
    β”‚   β”‚       β”œβ”€β”€ BUILD
    β”‚   β”‚       └── main.go
    β”‚   β”œβ”€β”€ go.mod
    β”‚   β”œβ”€β”€ go.sum
    β”‚   └── pkg
    β”œβ”€β”€ proto
    β”‚   └── auth_service
    β”‚       β”œβ”€β”€ BUILD
    β”‚       └── service.proto
    └── python
        β”œβ”€β”€ BUILD
        β”œβ”€β”€ auth_service_client
        β”‚   β”œβ”€β”€ BUILD
        β”‚   β”œβ”€β”€ __init__.py
        β”‚   └── client.py
        β”œβ”€β”€ auth_test_service
        β”‚   β”œβ”€β”€ BUILD
        β”‚   β”œβ”€β”€ __init__.py
        β”‚   └── main.py
        β”œβ”€β”€ requirements.lock
        └── requirements.txt
The only way to package
auth_service_client
is by creating a BUILD file in
src
or the root of the repo?
given that auth_service_client depends on generated proto files
That does seem to work, just slightly unfortunate that my
python_distribution
has to live outside the actual library code. Thank you!
e
The history is Twitter and bad practices in JVM land. Basically trying to prevent the same source getting published in >1 public artifact ever.
That happened there for internal distributions in spades and was a disaster as you might imagine. Multiple different versions of a source file, class file, what have you on the CLASSPATH, PYTHONPATH, etc leads to chaos.
h
In a nutshell: for each python source, Pants needs to choose a single
python_distribution
to publish it in. There is a documented algorithm for how it selects that
python_distribution
, and that algorithm relies on it being "above" the source file in the filesystem hierarchy.
l
That makes sense, thanks guys. Just not obvious how that applies to codegen’d code. Copy/pasting my
python_distribution
into a new BUILD file in the parent src directory and then adjusting the path to
:lib
worked exactly as expected.
I say non-obvious because it’s not clear to me where that python_source for the generated proto code ends up. It has the same source root as my
auth_service_client
project set, so I guess it ends up in a sibling directory there? Could I also have put this distribution in the BUILD file under
python
?