adamant-coat-28831
04/07/2023, 6:01 PMproject/
--A/
----lib_A/
------BUILD
----dist_A/
------BUILD
--shared_dist/
----BUILD
where the sources in project/A/lib_A
have a dependency on project/shared_dist
and project/A/dist_A
. The directory project/A/dist_A
builds a package that has 3rd party dependencies. (FYI, when this package runs, it loads an artifact that was produced by lib_A
- this is why the directories are grouped together in project/A
.) shared_dist
also builds a package that has 3rd party dependencies.
We are trying to reorganize the structure to the following:
project/
--A/
----dist_A/
----lib_A/
----BUILD
--shared_dist/
----source.py
----BUILD
where project/A/dist_A
now builds a package that contains the sources in dist_A
, the sources in lib_A
and the dependencies from shared_dist
. My first attempt of running ./pants package …
resulted in the error
NoOwnerError: No python_distribution target found to own project/A/shared/path/to/file.py:../lib@resolve=my_resolve. Note that the owner must be in or above the owned target's directory, and must depend on it (directly or indirectly).
It seems that the package is unable to pull in the resources in the shared_dist
package directory. The link to the documentation on python-distributions indicates this is possible, so I’m not sure where my mistake lies. Does anything jump out at you or do you need more information?happy-kitchen-89482
04/07/2023, 7:42 PMpython_distribution
in project/shared_dist/BUILD
(I assume) and project/A/BUILD
is not above it in the hierarchyshared_dist
to be used by multiple projects, A, B, C etc?adamant-coat-28831
04/07/2023, 7:45 PMhappy-kitchen-89482
04/07/2023, 7:45 PMpython_distribution
in project/shared_dist/BUILD
, that packages that code separately, and then generates requirements on it when you package Aadamant-coat-28831
04/07/2023, 7:48 PMhappy-kitchen-89482
04/07/2023, 7:50 PMrich-london-74860
04/07/2023, 7:50 PMshared_dist
be treated specially because it is also in pants
, or will it be treated like any other pypi dependency?happy-kitchen-89482
04/07/2023, 7:55 PMrich-london-74860
04/07/2023, 7:57 PMshared_dist
and A
must use separate resolves or else the contents of shared_dist
will conflict with the shared_dist
that’s a pip requirement in A
?happy-kitchen-89482
04/07/2023, 8:01 PMrich-london-74860
04/07/2023, 8:11 PMproject/shared_dist/BUILD
there is a python_sources
target generator with the name lib
and python_distribution
target with the name dist
• In project/A/lib_A/BUILD
there is python_sources
target generator with a dependency on project/shared_dist:lib
• In project/A/dist_A/BUILD
there is a python_requirements
target generator and a python_sources
target generator with a dependency on the python_requirements
.
With this change, in project/A/BUILD
we should:
• Add the name of the package built by project/shared_dist:dist
to the python_requirements
• Combine python_sources
targets for lib_A
and dist_A
with only a dependency
• Remove the dependency on project/shared_dist:lib
Is that right?project/shared_dist:dist
does not match the path shared_dist
, but I suppose that would be resolved by module_mapping
?happy-kitchen-89482
04/07/2023, 8:53 PMproject/shared_dist:lib
or the python_requirements
, they might be inferred from imports.rich-london-74860
04/07/2023, 9:06 PMyou don’t to do the things you mention in your second three bulletsDo you mean that we do not need to do all 3 of the 2nd set of 3 bullets? Or that we do not need to do the 2nd bullet of the 2nd set of 3 bullets (the 5th bullet)?
Combineregardless of any dependencies to put the contents oftargets forpython_sources
andlib_A
dist_A
lib_A
and dist_A
into the same python_distribution
Right now, we do not have this
Add the name of the package built byand I based on what you wrote above, adding this will fix our problem. @adamant-coat-28831 can you confirm? Are you saying that this is not necessary?to theproject/shared_dist:dist
python_requirements
Remove the dependency onproject/shared_dist:lib
happy-kitchen-89482
04/08/2023, 10:44 PMpython_sources
any way you like, but merging them makes sense, you do need one explicit dep from the python_distribution
to its entry point(s)python_requirements
or remove the project/shared_dist:lib
dep (although I would assume that dep would be inferred, so you can remove an explicit dep if so)python_requirements
is for third-party codetest
and run
package
will generate a requirement on it, assuming you will publish itrich-london-74860
04/17/2023, 3:52 PMBut you don’t want to add the package to theIt was necessary to remove theor remove thepython_requirements
depproject/shared_dist:lib
project/shared_dist:lib
dependency, that is what caused the original issue that @adamant-coat-28831 reported because project/shared_dist
is not a sub directory of project/A
We will also verify that we can remove the package name from python_requirements
you do need one explicit dep from theWhat if theto its entry point(s)python_distribution
python_distribution
package does not have an entry point? In our use case, we have a library with modules that are imported, there’s nothing that’s executed so there is no need for an entry point.happy-kitchen-89482
04/17/2023, 4:20 PMpython_distribution
to a set of sources from which all the sources you want to be packaged in the python_distribution can be reached via (inferred or explicit) depsrich-london-74860
04/19/2023, 3:47 AMresolve
One thing that we neglected to mention is the fact that our project uses multiple resolves and shared_dist
is used across multiple resolves. Therefore, the python_sources
target in project/shared_dist/BUILD
uses resolve=parameterize("my_resolve", "some_resolve", ...)
to be used across those different resolves. A
uses one particular resolve so the python_sources
target in project/A/BUILD
uses resolve="my_resolve"
. Since that’s a parameter for the python_sources
target in project/shared_dist/BUILD
, everything works.
However, the python_distribution
target does not have a resolve
parameter. In project/shared_dist/BUILD
the python_distribution
target must specify a dependency on python_source
targets that has a parametrized resolve. Which resolve should it use? Previously, we thought “I guess we should just pick one” and specified dependencies=[":lib@resolve=some_resolve"]
.
If you look back at the error that @adamant-coat-28831 originally posted:
NoOwnerError: No python_distribution target found to own project/A/shared/path/to/file.py:../lib@resolve=my_resolve. Note that the owner must be in or above the owned target's directory, and must depend on it (directly or indirectly).
This error was correct. There did not exist a python_distribution
target that owned project/A/shared/path/to/file.py:../lib@resolve=my_resolve
, but there was a python_distribution
target that owned project/A/shared/path/to/file.py:../lib@resolve=some_resolve
I have fixed this problem by duplicating the dependencies parameter in the python_distribution
target for every resolve parameter:
dependencies=[
":lib@resolve=my_resolve",
":lib@resolve=some_resolve",
...
]
Is this the correct way to do this?