https://pantsbuild.org/ logo
#general
Title
# general
c

clean-window-81064

02/06/2023, 9:30 AM
Hey All, Had a question - We have a monorepo structure & we were earlier packaging a single distribution which worked flawlessly, now we want to package another python-distribution from the same repo but when defining another
python_distribution
in the root I am observing
AmbiguousOwnerError
exception. I am quite certain that pants can package multiple distributions from the same repo, so maybe I am approaching this wrong ? The 2 distributions namely
rl
and
data_processing
share some common library code such as quitter, protos, dao etc and the expectation here is that they get packaged in both the distributions (i.e without a single distribution being the sole owner). I think in my case I essentially want the same code to be published in multiple distributions (the 2 distributions are isolated from each other and won’t ever run together so the associated risk is mitigated) Error snippet -
Copy code
AmbiguousOwnerError: Found multiple sibling python_distribution targets that are the closest ancestor dependees of src/backend/abc/libs/service/quitter.py:quitter and are therefore candidates to own it: src/backend/abc:rl, src/backend/abc:data_processing. Only a single such owner is allowed, to avoid ambiguity. See <https://www.pantsbuild.org/v2.14/docs/python-distributions> for how python_sources targets are mapped to distributions. See <https://www.pantsbuild.org/v2.14/docs/python-distributions>.
I couldn’t find a lot of information on pants docs related to
AmbiguousOwnerError
& was wondering if this is a loose requirement and maybe there is a flag to tweak it such that >=2 distributions can live together in the root ? or is there any other standard process for packaging multiple distributions. Another way I could think of is - Package the common code as separate distribution (say
common_libs
) and mark it as dependency in the final distributions i.e
rl
and
data_processing
or manual setup.py (which I am trying to avoid) PS: Fwiw, our near-future plan is to leverage the pants
python_distribution
target for packaging multiple packages.
r

refined-addition-53644

02/06/2023, 10:44 AM
https://www.pantsbuild.org/docs/python-distributions#mapping-source-files-to-distributions
Naively, you might think that a
python_distribution
publishes all the code of all the
python_source
targets it transitively depends on. But that could easily lead to trouble if you have multiple distributions that share common dependencies. You typically don’t want the same code published in multiple distributions, as this can lead to all sorts of runtime import issues.
If you use a handwritten
setup.py
, you have to figure this out for yourself - Pants will bundle whatever the script tells it to.
So this is intentional and the doc provides the explanation.
c

clean-window-81064

02/06/2023, 10:58 AM
Thanks @refined-addition-53644, I went through this while trying to understand how pants work for distributions. I was wondering if folks here had any ideas/suggestion on how to go around this (while trying to keep my monorepo future-proof from any potential pitfalls)? since this is a requirement in my case 😄
r

refined-addition-53644

02/06/2023, 11:03 AM
I am not entirely sure and would wait for input from someone from pants core team but one thing I see in the doc is that one distribution (D1) can depend on another (D2). So in your case you can package the common file only inside one of the distribution (D1) and then let pants figure it out how to package it for D2.
The generated
setup.py
will have its
install_requires
set to include the 3rdparty dependencies of the code bundled in the distribution, plus any other distributions from your own repo. For example, if distribution D1 contains code that has a dependency on some source file S, and that source file is published in distribution D2, then D1's requirements will include a dependency on D2. In other words, Pants does the right thing.
👍 1
c

cold-vr-15232

02/06/2023, 11:58 AM
Or if you don't want to have D1 to depend on D2, you can extract the common parts as a separate library, used by D1 and D2, and you don't have to publish it - pants will include it in D1 and D2.
c

clean-window-81064

02/06/2023, 12:14 PM
Yeah, this is indeed an option. But I will have to do a lot of changes for this since in the current setup we let pants auto-infer the dependencies when packaging targets, thus it’s automatically being inferred in both the places. 🤔
c

cold-vr-15232

02/06/2023, 2:35 PM
If you have the internal library available to pants, it will resolve it automatically. I don't know how your configuration is, so changes might need to be done.
c

clean-window-81064

02/07/2023, 10:15 AM
The internal libs (including the proto) are indeed available to pants & it is able to resolve automatically but when creating the
python_distribution
it’s throwing ambiguity error since the common libs are to be shared in both the distributions and pants doesn’t allow multiple owners for same python_source. So I am stuck now 😕
c

cold-vr-15232

02/07/2023, 1:03 PM
I may not be entirely accurate here, but I think that if you create a separate python_distribution target for the common parts, and refer to it from each of the other distribution, then it will work. I have a similar situation, and no problems with it.
👍 1
10 Views