Hi, I am trying to migrate from this project structure:

project/
--A/
----lib_A/
------BUILD
----dist_A/
------BUILD
--shared_dist/
----BUILD

where the sources in

project/A/lib_A

have a dependency on

project/shared_dist

and

project/A/dist_A

. The directory

project/A/dist_A

builds a package that has 3rd party dependencies. (FYI, when this package runs, it loads an artifact that was produced by

lib_A

- this is why the directories are grouped together in

project/A

.)

shared_dist

also builds a package that has 3rd party dependencies. We are trying to reorganize the structure to the following:

project/
--A/
----dist_A/
----lib_A/
----BUILD
--shared_dist/
----source.py
----BUILD

where

project/A/dist_A

now builds a package that contains the sources in

dist_A

, the sources in

lib_A

and the dependencies from

shared_dist

. My first attempt of running

./pants package …

resulted in the error

NoOwnerError: No python_distribution target found to own project/A/shared/path/to/file.py:../lib@resolve=my_resolve. Note that the owner must be in or above the owned target's directory, and must depend on it (directly or indirectly).

It seems that the package is unable to pull in the resources in the

shared_dist

package directory. The link to the documentation on python-distributions indicates this is possible, so I’m not sure where my mistake lies. Does anything jump out at you or do you need more information?

I think the error message is telling you the issue. Pants associates source files to the distribution that publishes them using this algorithm: https://www.pantsbuild.org/docs/python-distributions#mapping-source-files-to-distributions

yes, that is the intention

You don't want to package it in A, and in B, and in C, and so on

do you mean to include the shared package in the requirements for package A?

Yes, exactly

In this case, will

shared_dist

be treated specially because it is also in

pants

, or will it be treated like any other pypi dependency?

When you publish A, Pants will figure out by following deps that A's wheel must have a requirement on shared_dist's wheel, and add one to the list of A's requirements

But doesn’t that also mean that if we’re testing A, we’re using the code that’s been deployed, not the code in the repo?

No, this is only when building wheels

Right now: • In

project/shared_dist/BUILD

there is a

python_sources

target generator with the name

lib

and

python_distribution

target with the name

dist

• In

project/A/lib_A/BUILD

there is

python_sources

target generator with a dependency on

project/shared_dist:lib

• In

project/A/dist_A/BUILD

there is a

python_requirements

target generator and a

python_sources

target generator with a dependency on the

python_requirements

. With this change, in

project/A/BUILD

we should: • Add the name of the package built by

project/shared_dist:dist

to the

python_requirements

• Combine

python_sources

targets for

lib_A

and

dist_A

with only a dependency • Remove the dependency on

project/shared_dist:lib

Is that right?

It's simpler than that - you don't to do the things you mention in your second three bullets. Pants does the right thing automatically both when testing and when packaging. Also, you don't necessarily need to add the explicit deps on

project/shared_dist:lib

or the

python_requirements

, they might be inferred from imports.

you don’t to do the things you mention in your second three bullets

Do you mean that we do not need to do all 3 of the 2nd set of 3 bullets? Or that we do not need to do the 2nd bullet of the 2nd set of 3 bullets (the 5th bullet)?

You don't need to do any of them

But you don’t want to add the package to the

python_requirements

or remove the

project/shared_dist:lib

dep

It was necessary to remove the

project/shared_dist:lib

dependency, that is what caused the original issue that @adamant-coat-28831 reported because

project/shared_dist

is not a sub directory of

project/A

We will also verify that we can remove the package name from

python_requirements

Yeah, what you actually need is explicit dependencies from the

python_distribution

to a set of sources from which all the sources you want to be packaged in the python_distribution can be reached via (inferred or explicit) deps

Ok, I now fully understand our problem. The problem is related to

resolve

One thing that we neglected to mention is the fact that our project uses multiple resolves and

shared_dist

is used across multiple resolves. Therefore, the

python_sources

target in

project/shared_dist/BUILD

uses

resolve=parameterize("my_resolve", "some_resolve", ...)

to be used across those different resolves.

A

uses one particular resolve so the

python_sources

target in

project/A/BUILD

uses

resolve="my_resolve"

. Since that’s a parameter for the

python_sources

target in

project/shared_dist/BUILD

, everything works. However, the

python_distribution

target does not have a

resolve

parameter. In

project/shared_dist/BUILD

the

python_distribution

target must specify a dependency on

python_source

targets that has a parametrized resolve. Which resolve should it use? Previously, we thought “I guess we should just pick one” and specified

dependencies=[":lib@resolve=some_resolve"]

. If you look back at the error that @adamant-coat-28831 originally posted:

NoOwnerError: No python_distribution target found to own project/A/shared/path/to/file.py:../lib@resolve=my_resolve. Note that the owner must be in or above the owned target's directory, and must depend on it (directly or indirectly).

This error was correct. There did not exist a

python_distribution

target that owned

project/A/shared/path/to/file.py:../lib@resolve=my_resolve

, but there was a

python_distribution

target that owned

project/A/shared/path/to/file.py:../lib@resolve=some_resolve

I have fixed this problem by duplicating the dependencies parameter in the

python_distribution

target for every resolve parameter:

dependencies=[
    ":lib@resolve=my_resolve",
    ":lib@resolve=some_resolve",
    ...
]

Is this the correct way to do this?