https://pantsbuild.org/ logo
Join SlackCommunities
Powered by
```NoOwnerError: No python_distribution target fou...
# general
b
Copy code
NoOwnerError: No python_distribution target found to own gh/util/file_utils.py. Note that the owner must be in or above the owned target's directory, and must depend on it (directly or indirectly). See <https://www.pantsbuild.org/v2.0/docs/python-setup-py-goal> for how python_library targets are mapped to distributions.
a
i expect this is likely the result of dependency inference. could you confirm whether you 
import gh.util.file_utils
(or 
from gh.util.file_utils import xxx
) anywhere in that target?
b
how many layers of transitive count?
a
and this part of that docs page should have more info on why it needs to be a 
python_distribution()
too, which is likely more relevant:
Copy code
So in practice, multiple libraries are typically bundled into a single distribution. Naively, you might think that a python_distribution publishes all the code of all the python_library targets it transitively depends on. But that could easily lead to trouble if you have multiple distributions that share common dependencies. You typically don't want the same code published in multiple distributions, as this can lead to all sorts of runtime import issues.

Instead, Pants applies the following algorithm:
b
yeah, I read that, but honestly, you could use more concrete examples in there 🙂
a
thank you! i also found it somewhat difficult to parse when reading it just now
b
as to the first Q - gh/leader_election/* def does NOT refer in any way to 
file_utils
a
i believe that this paragraph is key:
If there are multiple such exported libraries at the same degree of ancestry, the ownership
is ambiguous and an error is raised. If there is no python_distribution that depends on L
and is its ancestor, then there is no owner and an error is raised.
we are seeing "an error is raised" right now.
so i think that some 
python_distribution()
target (which i think is a transitive dep of 
gh/leader_election/*
) depends on 
file_utils
so i would use 
./pants dependencies
and 
rg file_utils
to see which files import 
file_utils
, and see where they intersect with the dependencies of the target you are trying to 
setup-py
is that clear?
rg
is ripgrep, you can use ack or ag or anything else too
b
git grep file_utils
shows me the answer about who imports it
a
even better, great
b
I can test your theory by commenting out the only other BUILD::python_distribution in the repo, I think
👍🏻 1
ahhhh
⁉️ 1
yeah, so what I tried to build deps on 'gh/core' (which should exist as a diff pip mod to install - and be brought in by the one I'm trying to build)
👏🏻 1
so when I comment out python_distribution in gh/core/BUILD....
boom
❤️ 1
ResolveError: 'core' was not found in namespace 'gh/core'.
so...
that kinda makes sense
but why complain about a file in a completely different part fo the tree?
⁉️ 1
a
that is a good question
b
it complains about 
gh/util/file_utils.py
and util is not a dep of either core nor leader_electtion
h
ResolveError: ‘core’ was not found in namespace ‘gh/core’.
What’s the full message? Usually a ResolveError is when you use an address Pants doesn’t know about, and it will suggest some alternatives For example, if you mistype 
:my_tgt
to be 
:my_tg
✍🏻 1
a
it was a 
NoOwnerError
i think?
b
taht's if I commented out the python_distribution() in gh/core/BUILD
👍 1
which is not an expected state of thigns
h
it complains about gh/util/file_utils.py
Is anyone importing 
file_utils.py
? 
./pants dependees gh/util/file_utils.py
☝🏻 1
a
ahhhh i didn't think of that
h
Oh sorry I misread that you said something already does. But it would be helpful to see who Pants thinks is importing it by running 
./pants dependees
b
well, I used 
git grep
but yes
Copy code
$ git grep file_util
util/object_input.py:from gh.util.file_utils import load_list_from_file
Copy code
./pants dependees gh/util/file_utils.py 
Scrubbed PYTHONPATH=/home/mfischer/src/ghpylibs/python from the environment.
16:37:30.27 [WARN] <unknown>:56: DeprecationWarning: invalid escape sequence \.
16:37:30.27 [WARN] <unknown>:57: DeprecationWarning: invalid escape sequence \.
16:37:32.50 [WARN] <unknown>:587: DeprecationWarning: invalid escape sequence \{
16:37:32.52 [WARN] <unknown>:56: DeprecationWarning: invalid escape sequence \.
16:37:32.52 [WARN] <unknown>:57: DeprecationWarning: invalid escape sequence \.
16:37:32.80 [WARN] <unknown>:587: DeprecationWarning: invalid escape sequence \{
16:37:33.26 [WARN] <unknown>:614: DeprecationWarning: invalid escape sequence \w
16:37:34.20 [WARN] <unknown>:614: DeprecationWarning: invalid escape sequence \w
gh/util
gh/util/object_input.py
👍 1
that bug seems still to be present 🙂
should I reinstall pants? 😛
h
Are you using 2.0.0b2? Change 
pants_version
in 
pants.toml
👖 1
b
pants_version = "2.0.0b0"
if I change it there, will it "auto update" or...?
h
Yep 🙂 the 
./pants
bash script reads the value and auto-updates for you.
b
seems like it is...
Copy code
16:40:55.45 [WARN] gh/garcon/selfserviceauthorization/model.py:614: DeprecationWarning: invalid escape sequence \w
  _re12 = re.compile("^([01]?[0-9]):[0-5][0-9](\w{1,2})?$")
well, you got the filenames in there
🔥 1
(begs the question why parsing 
import
statements should show those warnings but...)
👀 1
h
Yeah. It’s not threadsafe for us to capture the warnings, due to a Python limitation: https://github.com/pantsbuild/pants/pull/10857 I wanted to capture warnings instead
b
in any case, pants dependees is showing me what 
git grep
did
FWIW, the upgrade did not change my original problem - it still complains about 'No python_distribution target found to own gh/util/file_utils.py'
w
@billowy-motherboard-58443: sorry for the trouble, but would you mind filing a ticket explaining the whole repro for this?
b
I'd have to invent a POC repo as well, I suspect...
a
you could zip it up instead of making a new repo i think
w
yea, i hope not… it’s possible that just the file layout/names are enough?
☝🏻 1
a
what stu said
b
I can't zip it, I'm working off our "real code"
👍🏻 1
this is a "migrate one big lib to a monorepo of libs" project
sussing out suitablility of pants for it
w
thanks a lot for the assistance in polishing things. sorry that you ran into this.
bringing the folks who are most familiar with the setup-py ownership code up to speed would be easiest with a sketch of the example. if you don’t have time, i can give it a shot.
👍 2
b
yeah, I'm trying to create a miniature of the repo, with all the "real" code removed
and see if it still has the bug
bug/issue
w
thanks a lot!
b
Not repro-ing....
w
which target owns 
gh/util/file_utils.py
?
not 
*_distribution
, but 
*_library
./pants dependees gh/util/file_utils.py
should report it
b
hmmm
we did this above but
Copy code
gh/util:util_lib
gh/util/object_input.py:util_lib
w
ok. and your 
*_distribution
is located where?
b
so... I just went through and made sure that each of the BUILD files had its python_library() and python_distribution() with unique names in each
w
the same directory?
b
and... problem disappeared?????
w
it’s possible that there was no distribution that depended on that file (transitively) and lived above it in the directory structure
but yea, sorry for the trouble. the ratios we’re used to seeing on libraries vs distributions is hundreds of libraries/modules per distribution, so you’re dealing with an unusual amount of boilerplate
b
does the 'inference' extend to 3rd party (e.g. pypi) libs?
w
yes.
b
ok, so I did setup-py and a file does 
import casssndra
cassandra-driver does not make it do the setup.py generated file
w
see the 
module_mapping
on https://www.pantsbuild.org/docs/python-third-party-dependencies … but you might not need a mapping if the requirement names match the imports
mm. so yea, see that link then ^
b
I want to edit how your docs left-gutter nav works.....
requirements.txt + "//:name" syntax?
h
I want to edit how your docs left-gutter nav works.....
We use readme.io to host our docs. I don’t think we can edit this easily
w
@billowy-motherboard-58443: no, the 
module_mapping
aspect in particular: can find in page
b
I don't think that's my problem ATM
the bit right after that with psycopg2 is more what I was looking for? 🙂
w
@billowy-motherboard-58443: if the requirement name is 
cassandra-driver
but the import is 
cassandra
, then it’s likely that it will not be inferred
b
I'm guessing the module_mapping is about avoiding collisions?
ahhhhh
the way the doc is written, it seems like it is intended to address collisions
w
i’ll add an aside there
b
is the intent
"published-name": "name-one-imports" ?
👍 1
w
yep
too many names for each of those things i suppose 😃
i added one additional combination to the page
b
ok, I used an explicit "//:cassandra-driver" in python_distribution(dependencies=)
which got it into the setup.py
but it ignored the version I had in constraints.txt
it had no version at all in setup.py
w
it will be the version from your requirements.txt
but note also that putting the dependency in the distribution is the wrong place
if your code depends on that requirement, the code should have the dependency.
b
better in the python_library() ?
w
if you intend to use inference, would suggest fixing the mapping so that you don’t need to manually add the dep.
yes
b
kk
whooop
as soon as I did that, it complained about no separate requirements.txt in that lib's subtree
I had the impression only 1 top-level requirements.txt was used....
w
you can have as many as you like, but we recommend one for now, yea
the BUILD content (
python_requirements(..)
) should be next to the requirements.txt, wherever it is.
h
What’s the error message? (It’s helpful for us to always include the original error message for things. If it’s huge, Slack’s code snippet feature is helpful, or Gists/Pastebin)
b
Copy code
MappingError: Failed to parse gh/leader_election/BUILD:
[Errno 2] No such file or directory: '/src/gh-pants-test/gh/leader_election/requirements.txt'
top of tree is gh-pants-test
👍 1
which is where requirements.txt and constraints.txt live
w
the 
python_requirements(..)
definition
b
so move the python_requirements() to the top level build, rather than the one with the python_lib and python_dist declarations for the particular pkg I'm trying to build?
👍 1
sort of got it, but still getting no version in the setup.py
(for cassandra-driver
it doesn't check constraints.txt for that?
h
For the file that imports 
cassandra
, you can run 
./pants dependencies path/to/file.py
to check that 
//:cassandra-driver
is being properly inferred
b
yes, it gets that
but my problem is this part of the generated setup.py
Copy code
'install_requires': (
        'cassandra-driver',
        'gh.config==0.1',
        'gh.core==0.1',
        'gh.util==0.1',
    ),
no version for cassandra-driver
where I do have it set in constraints.txt
(but no where else)
h
it doesn’t check constraints.txt for that?
Not for dependency inference. The 
python_requirements()
target is a macro that reads your 
requirements.txt
and creates a bunch of 
python_requirement_library
targets for you automatically. One target per each distinct requirement. Dependency inference uses this when analyzing your imports. Meanwhile, the 
constraints.txt
is solely used by Pex (which uses pip) when resolving requirements.
b
kk
and yes, just proved to myself that that works
my top level BUILD is now
Copy code
python_requirements()

python_requirement_library(
    name="cassandra",
    requirements=["cassandra-driver==3.24"],
    module_mapping={
        "cassandra-driver": ["cassandra"],
    },
)
`
h
Oh, okay, thanks for including that snippet. If you want to constrain 
cassandra-driver
in your generated 
setup.py
, then you need the constraint that you want in 
requirements.txt
We very intentionally do not use your constraints.txt when generating setup.py. You might want your setup.py to be loose, like 
>=3.0,<4
, whereas a 
constraints.txt
should be pinning to a specific version like 
==3.3
b
does that seem sensible?
oh, so it WILL use reruirements.txt in generating setup.py?
h
That’s fine to do it that way too, if you don’t want to have 
cassandra-driver
in your 
requirements.txt
. If it’s okay to be in your 
requirements.txt
, that’s simpler than explicitly defining a 
python_requirement_library
You would update your 
requirmenets.txt
to be 
cassandra-driver==3.24
, whereas right now it sounds like it’s 
cassandra-driver
b
hmm, seems not to
I just tried with taking the vesion out of of python_requirement_library and put it in requirements.txt
and it does NOT get included in setup.py
h
To summarize, I’d recommend: 1. move the 
module_mapping
to your 
python_requirements()
macro 2. delete the 
python_requirement_library
target. This is important for dependency inference to work, otherwise you will have two targets describing the same requirement, and dep inference won’t know what to do with the ambiguity 3. Update your 
requirements.txt
to use 
==3.3
b
3 didn't help - not reflected in the generated setup.py
I can try 1 &2
h
Did you first do #1 and #2?
b
an no
ah, ok, worked
top level BUILD is now
Copy code
python_requirements(
    # mapping for "pip install name is different from import name"
    module_mapping={
        "cassandra-driver": ["cassandra"],
    },
)
💯 1
and that's it
h
I suspect what happened is that if you didn’t first do #1, then dependency inference will not end up using what’s defined in your 
requriements.txt
. Pants doesn’t know that 
import cassandra
means 
cassandra-driver
. So updating 
requirements.txt
doesn’t do anything. But, it does know about the inline target you created in the BUILD file, which is why the value is still showing up. -- Now, once you do #1 but not yet #2, you have the issue that you now have two distinct targets that are both mapped to the import 
cassandra
. This results in neither target being used, and nothing showing up at all. Dependency inference cannot disambiguate what you want. -- It requires all 3 steps to work properly.
Awesome!
27 Views
Open in Slack
PreviousNext
Powered by