https://pantsbuild.org/ logo
#general
Title
# general
a

adamant-coat-28831

04/07/2023, 6:01 PM
Hi, I am trying to migrate from this project structure:
Copy code
project/
--A/
----lib_A/
------BUILD
----dist_A/
------BUILD
--shared_dist/
----BUILD
where the sources in
project/A/lib_A
have a dependency on
project/shared_dist
and
project/A/dist_A
. The directory
project/A/dist_A
builds a package that has 3rd party dependencies. (FYI, when this package runs, it loads an artifact that was produced by
lib_A
- this is why the directories are grouped together in
project/A
.)
shared_dist
also builds a package that has 3rd party dependencies. We are trying to reorganize the structure to the following:
Copy code
project/
--A/
----dist_A/
----lib_A/
----BUILD
--shared_dist/
----source.py
----BUILD
where
project/A/dist_A
now builds a package that contains the sources in
dist_A
, the sources in
lib_A
and the dependencies from
shared_dist
. My first attempt of running
./pants package …
resulted in the error
Copy code
NoOwnerError: No python_distribution target found to own project/A/shared/path/to/file.py:../lib@resolve=my_resolve. Note that the owner must be in or above the owned target's directory, and must depend on it (directly or indirectly).
It seems that the package is unable to pull in the resources in the
shared_dist
package directory. The link to the documentation on python-distributions indicates this is possible, so I’m not sure where my mistake lies. Does anything jump out at you or do you need more information?
h

happy-kitchen-89482

04/07/2023, 7:42 PM
I think the error message is telling you the issue. Pants associates source files to the distribution that publishes them using this algorithm: https://www.pantsbuild.org/docs/python-distributions#mapping-source-files-to-distributions
There must be a python_distribution target that (transitively) depends on that code, and is at or above it in the hierarchy
In your case there is no
python_distribution
in
project/shared_dist/BUILD
(I assume) and
project/A/BUILD
is not above it in the hierarchy
But here's the thing - are you intending for
shared_dist
to be used by multiple projects, A, B, C etc?
If so then you will want to publish it separately (as its name implies)
a

adamant-coat-28831

04/07/2023, 7:45 PM
yes, that is the intention
h

happy-kitchen-89482

04/07/2023, 7:45 PM
You don't want to package it in A, and in B, and in C, and so on
it is a very bad idea to have the same code published in multiple dists
so probably what you want is a
python_distribution
in
project/shared_dist/BUILD
, that packages that code separately, and then generates requirements on it when you package A
a

adamant-coat-28831

04/07/2023, 7:48 PM
do you mean to include the shared package in the requirements for package A?
h

happy-kitchen-89482

04/07/2023, 7:50 PM
Yes, exactly
Pants will generate that requirement when it generates A's setup.py
because it will see the dep
r

rich-london-74860

04/07/2023, 7:50 PM
In this case, will
shared_dist
be treated specially because it is also in
pants
, or will it be treated like any other pypi dependency?
h

happy-kitchen-89482

04/07/2023, 7:55 PM
When you publish A, Pants will figure out by following deps that A's wheel must have a requirement on shared_dist's wheel, and add one to the list of A's requirements
but it expects that shared_dist's wheel has been published like any other pypi dependency
👀 1
So it's not treated specially in the metadata
r

rich-london-74860

04/07/2023, 7:57 PM
But doesn’t that also mean that if we’re testing A, we’re using the code that’s been deployed, not the code in the repo?
And that
shared_dist
and
A
must use separate resolves or else the contents of
shared_dist
will conflict with the
shared_dist
that’s a pip requirement in
A
?
h

happy-kitchen-89482

04/07/2023, 8:01 PM
No, this is only when building wheels
when running tests, it all runs directly from source
r

rich-london-74860

04/07/2023, 8:11 PM
Right now: • In
project/shared_dist/BUILD
there is a
python_sources
target generator with the name
lib
and
python_distribution
target with the name
dist
• In
project/A/lib_A/BUILD
there is
python_sources
target generator with a dependency on
project/shared_dist:lib
• In
project/A/dist_A/BUILD
there is a
python_requirements
target generator and a
python_sources
target generator with a dependency on the
python_requirements
. With this change, in
project/A/BUILD
we should: • Add the name of the package built by
project/shared_dist:dist
to the
python_requirements
• Combine
python_sources
targets for
lib_A
and
dist_A
with only a dependency • Remove the dependency on
project/shared_dist:lib
Is that right?
I should also note that the package name for
project/shared_dist:dist
does not match the path
shared_dist
, but I suppose that would be resolved by
module_mapping
?
h

happy-kitchen-89482

04/07/2023, 8:53 PM
It's simpler than that - you don't to do the things you mention in your second three bullets. Pants does the right thing automatically both when testing and when packaging. Also, you don't necessarily need to add the explicit deps on
project/shared_dist:lib
or the
python_requirements
, they might be inferred from imports.
r

rich-london-74860

04/07/2023, 9:06 PM
you don’t to do the things you mention in your second three bullets
Do you mean that we do not need to do all 3 of the 2nd set of 3 bullets? Or that we do not need to do the 2nd bullet of the 2nd set of 3 bullets (the 5th bullet)?
We need to do this:
Combine
python_sources
targets for
lib_A
and
dist_A
regardless of any dependencies to put the contents of
lib_A
and
dist_A
into the same
python_distribution
Right now, we do not have this
Add the name of the package built by
project/shared_dist:dist
to the
python_requirements
and I based on what you wrote above, adding this will fix our problem. @adamant-coat-28831 can you confirm? Are you saying that this is not necessary?
Remove the dependency on
project/shared_dist:lib
h

happy-kitchen-89482

04/08/2023, 10:44 PM
You don't need to do any of them
You can organize your
python_sources
any way you like, but merging them makes sense, you do need one explicit dep from the
python_distribution
to its entry point(s)
But you don't want to add the package to the
python_requirements
or remove the
project/shared_dist:lib
dep (although I would assume that dep would be inferred, so you can remove an explicit dep if so)
python_requirements
is for third-party code
this is first-party code, that you also happen to publish
but it will be consumed directly from the repo in
test
and
run
And
package
will generate a requirement on it, assuming you will publish it
r

rich-london-74860

04/17/2023, 3:52 PM
But you don’t want to add the package to the
python_requirements
or remove the
project/shared_dist:lib
dep
It was necessary to remove the
project/shared_dist:lib
dependency, that is what caused the original issue that @adamant-coat-28831 reported because
project/shared_dist
is not a sub directory of
project/A
We will also verify that we can remove the package name from
python_requirements
you do need one explicit dep from the
python_distribution
to its entry point(s)
What if the
python_distribution
package does not have an entry point? In our use case, we have a library with modules that are imported, there’s nothing that’s executed so there is no need for an entry point.
h

happy-kitchen-89482

04/17/2023, 4:20 PM
Yeah, what you actually need is explicit dependencies from the
python_distribution
to a set of sources from which all the sources you want to be packaged in the python_distribution can be reached via (inferred or explicit) deps
1
An entry point is typically such a source - everything it needs is reachable by deps, so if you depend on it, you depend on them too.
But if you have no entry point, you'll need to find some other set of sources that can reach all the other ones
r

rich-london-74860

04/19/2023, 3:47 AM
Ok, I now fully understand our problem. The problem is related to
resolve
One thing that we neglected to mention is the fact that our project uses multiple resolves and
shared_dist
is used across multiple resolves. Therefore, the
python_sources
target in
project/shared_dist/BUILD
uses
resolve=parameterize("my_resolve", "some_resolve", ...)
to be used across those different resolves.
A
uses one particular resolve so the
python_sources
target in
project/A/BUILD
uses
resolve="my_resolve"
. Since that’s a parameter for the
python_sources
target in
project/shared_dist/BUILD
, everything works. However, the
python_distribution
target does not have a
resolve
parameter. In
project/shared_dist/BUILD
the
python_distribution
target must specify a dependency on
python_source
targets that has a parametrized resolve. Which resolve should it use? Previously, we thought “I guess we should just pick one” and specified
dependencies=[":lib@resolve=some_resolve"]
. If you look back at the error that @adamant-coat-28831 originally posted:
Copy code
NoOwnerError: No python_distribution target found to own project/A/shared/path/to/file.py:../lib@resolve=my_resolve. Note that the owner must be in or above the owned target's directory, and must depend on it (directly or indirectly).
This error was correct. There did not exist a
python_distribution
target that owned
project/A/shared/path/to/file.py:../lib@resolve=my_resolve
, but there was a
python_distribution
target that owned
project/A/shared/path/to/file.py:../lib@resolve=some_resolve
I have fixed this problem by duplicating the dependencies parameter in the
python_distribution
target for every resolve parameter:
Copy code
dependencies=[
    ":lib@resolve=my_resolve",
    ":lib@resolve=some_resolve",
    ...
]
Is this the correct way to do this?