https://pantsbuild.org/ logo
#general
Title
# general
b

billowy-motherboard-58443

10/01/2020, 4:21 PM
Copy code
NoOwnerError: No python_distribution target found to own gh/util/file_utils.py. Note that the owner must be in or above the owned target's directory, and must depend on it (directly or indirectly). See <https://www.pantsbuild.org/v2.0/docs/python-setup-py-goal> for how python_library targets are mapped to distributions.
a

aloof-angle-91616

10/01/2020, 4:22 PM
i expect this is likely the result of dependency inference. could you confirm whether you
import gh.util.file_utils
(or
from gh.util.file_utils import xxx
) anywhere in that target?
b

billowy-motherboard-58443

10/01/2020, 4:23 PM
how many layers of transitive count?
a

aloof-angle-91616

10/01/2020, 4:24 PM
and this part of that docs page should have more info on why it needs to be a
python_distribution()
too, which is likely more relevant:
Copy code
So in practice, multiple libraries are typically bundled into a single distribution. Naively, you might think that a python_distribution publishes all the code of all the python_library targets it transitively depends on. But that could easily lead to trouble if you have multiple distributions that share common dependencies. You typically don't want the same code published in multiple distributions, as this can lead to all sorts of runtime import issues.

Instead, Pants applies the following algorithm:
b

billowy-motherboard-58443

10/01/2020, 4:25 PM
yeah, I read that, but honestly, you could use more concrete examples in there 🙂
a

aloof-angle-91616

10/01/2020, 4:25 PM
thank you! i also found it somewhat difficult to parse when reading it just now
b

billowy-motherboard-58443

10/01/2020, 4:25 PM
as to the first Q - gh/leader_election/* def does NOT refer in any way to
file_utils
a

aloof-angle-91616

10/01/2020, 4:27 PM
i believe that this paragraph is key:
If there are multiple such exported libraries at the same degree of ancestry, the ownership
is ambiguous and an error is raised. If there is no python_distribution that depends on L
and is its ancestor, then there is no owner and an error is raised.
we are seeing "an error is raised" right now.
so i think that some
python_distribution()
target (which i think is a transitive dep of
gh/leader_election/*
) depends on
file_utils
so i would use
./pants dependencies
and
rg file_utils
to see which files import
file_utils
, and see where they intersect with the dependencies of the target you are trying to
setup-py
is that clear?
rg
is ripgrep, you can use ack or ag or anything else too
b

billowy-motherboard-58443

10/01/2020, 4:29 PM
git grep file_utils
shows me the answer about who imports it
a

aloof-angle-91616

10/01/2020, 4:30 PM
even better, great
b

billowy-motherboard-58443

10/01/2020, 4:30 PM
I can test your theory by commenting out the only other BUILD::python_distribution in the repo, I think
👍🏻 1
ahhhh
⁉️ 1
yeah, so what I tried to build deps on 'gh/core' (which should exist as a diff pip mod to install - and be brought in by the one I'm trying to build)
👏🏻 1
so when I comment out python_distribution in gh/core/BUILD....
boom
❤️ 1
ResolveError: 'core' was not found in namespace 'gh/core'.
so...
that kinda makes sense
but why complain about a file in a completely different part fo the tree?
⁉️ 1
a

aloof-angle-91616

10/01/2020, 4:32 PM
that is a good question
b

billowy-motherboard-58443

10/01/2020, 4:33 PM
it complains about
gh/util/file_utils.py
and util is not a dep of either core nor leader_electtion
h

hundreds-father-404

10/01/2020, 4:33 PM
ResolveError: ‘core’ was not found in namespace ‘gh/core’.
What’s the full message? Usually a ResolveError is when you use an address Pants doesn’t know about, and it will suggest some alternatives For example, if you mistype
:my_tgt
to be
:my_tg
✍🏻 1
a

aloof-angle-91616

10/01/2020, 4:33 PM
it was a
NoOwnerError
i think?
b

billowy-motherboard-58443

10/01/2020, 4:34 PM
taht's if I commented out the python_distribution() in gh/core/BUILD
👍 1
which is not an expected state of thigns
h

hundreds-father-404

10/01/2020, 4:35 PM
it complains about gh/util/file_utils.py
Is anyone importing
file_utils.py
?
./pants dependees gh/util/file_utils.py
☝🏻 1
a

aloof-angle-91616

10/01/2020, 4:35 PM
ahhhh i didn't think of that
h

hundreds-father-404

10/01/2020, 4:36 PM
Oh sorry I misread that you said something already does. But it would be helpful to see who Pants thinks is importing it by running
./pants dependees
b

billowy-motherboard-58443

10/01/2020, 4:36 PM
well, I used
git grep
but yes
Copy code
$ git grep file_util
util/object_input.py:from gh.util.file_utils import load_list_from_file
Copy code
./pants dependees gh/util/file_utils.py 
Scrubbed PYTHONPATH=/home/mfischer/src/ghpylibs/python from the environment.
16:37:30.27 [WARN] <unknown>:56: DeprecationWarning: invalid escape sequence \.
16:37:30.27 [WARN] <unknown>:57: DeprecationWarning: invalid escape sequence \.
16:37:32.50 [WARN] <unknown>:587: DeprecationWarning: invalid escape sequence \{
16:37:32.52 [WARN] <unknown>:56: DeprecationWarning: invalid escape sequence \.
16:37:32.52 [WARN] <unknown>:57: DeprecationWarning: invalid escape sequence \.
16:37:32.80 [WARN] <unknown>:587: DeprecationWarning: invalid escape sequence \{
16:37:33.26 [WARN] <unknown>:614: DeprecationWarning: invalid escape sequence \w
16:37:34.20 [WARN] <unknown>:614: DeprecationWarning: invalid escape sequence \w
gh/util
gh/util/object_input.py
👍 1
that bug seems still to be present 🙂
should I reinstall pants? 😛
h

hundreds-father-404

10/01/2020, 4:38 PM
Are you using 2.0.0b2? Change
pants_version
in
pants.toml
👖 1
b

billowy-motherboard-58443

10/01/2020, 4:39 PM
pants_version = "2.0.0b0"
if I change it there, will it "auto update" or...?
h

hundreds-father-404

10/01/2020, 4:39 PM
Yep 🙂 the
./pants
bash script reads the value and auto-updates for you.
b

billowy-motherboard-58443

10/01/2020, 4:39 PM
seems like it is...
Copy code
16:40:55.45 [WARN] gh/garcon/selfserviceauthorization/model.py:614: DeprecationWarning: invalid escape sequence \w
  _re12 = re.compile("^([01]?[0-9]):[0-5][0-9](\w{1,2})?$")
well, you got the filenames in there
🔥 1
(begs the question why parsing
import
statements should show those warnings but...)
👀 1
h

hundreds-father-404

10/01/2020, 4:42 PM
Yeah. It’s not threadsafe for us to capture the warnings, due to a Python limitation: https://github.com/pantsbuild/pants/pull/10857 I wanted to capture warnings instead
b

billowy-motherboard-58443

10/01/2020, 4:42 PM
in any case, pants dependees is showing me what
git grep
did
FWIW, the upgrade did not change my original problem - it still complains about 'No python_distribution target found to own gh/util/file_utils.py'
w

witty-crayon-22786

10/01/2020, 4:51 PM
@billowy-motherboard-58443: sorry for the trouble, but would you mind filing a ticket explaining the whole repro for this?
b

billowy-motherboard-58443

10/01/2020, 4:51 PM
I'd have to invent a POC repo as well, I suspect...
a

aloof-angle-91616

10/01/2020, 4:52 PM
you could zip it up instead of making a new repo i think
w

witty-crayon-22786

10/01/2020, 4:52 PM
yea, i hope not… it’s possible that just the file layout/names are enough?
☝🏻 1
a

aloof-angle-91616

10/01/2020, 4:52 PM
what stu said
b

billowy-motherboard-58443

10/01/2020, 4:52 PM
I can't zip it, I'm working off our "real code"
👍🏻 1
this is a "migrate one big lib to a monorepo of libs" project
sussing out suitablility of pants for it
w

witty-crayon-22786

10/01/2020, 4:54 PM
thanks a lot for the assistance in polishing things. sorry that you ran into this.
bringing the folks who are most familiar with the setup-py ownership code up to speed would be easiest with a sketch of the example. if you don’t have time, i can give it a shot.
👍 2
b

billowy-motherboard-58443

10/01/2020, 5:01 PM
yeah, I'm trying to create a miniature of the repo, with all the "real" code removed
and see if it still has the bug
bug/issue
w

witty-crayon-22786

10/01/2020, 5:02 PM
thanks a lot!
b

billowy-motherboard-58443

10/01/2020, 5:17 PM
Not repro-ing....
w

witty-crayon-22786

10/01/2020, 5:18 PM
which target owns
gh/util/file_utils.py
?
not
*_distribution
, but
*_library
./pants dependees gh/util/file_utils.py
should report it
b

billowy-motherboard-58443

10/01/2020, 5:20 PM
hmmm
we did this above but
Copy code
gh/util:util_lib
gh/util/object_input.py:util_lib
w

witty-crayon-22786

10/01/2020, 5:21 PM
ok. and your
*_distribution
is located where?
the same directory?
b

billowy-motherboard-58443

10/01/2020, 5:21 PM
so... I just went through and made sure that each of the BUILD files had its python_library() and python_distribution() with unique names in each
and... problem disappeared?????
w

witty-crayon-22786

10/01/2020, 5:23 PM
it’s possible that there was no distribution that depended on that file (transitively) and lived above it in the directory structure
but yea, sorry for the trouble. the ratios we’re used to seeing on libraries vs distributions is hundreds of libraries/modules per distribution, so you’re dealing with an unusual amount of boilerplate
b

billowy-motherboard-58443

10/01/2020, 5:24 PM
does the 'inference' extend to 3rd party (e.g. pypi) libs?
w

witty-crayon-22786

10/01/2020, 5:24 PM
yes.
b

billowy-motherboard-58443

10/01/2020, 5:24 PM
ok, so I did setup-py and a file does
import casssndra
cassandra-driver does not make it do the setup.py generated file
w

witty-crayon-22786

10/01/2020, 5:25 PM
see the
module_mapping
on https://www.pantsbuild.org/docs/python-third-party-dependencies … but you might not need a mapping if the requirement names match the imports
mm. so yea, see that link then ^
b

billowy-motherboard-58443

10/01/2020, 5:26 PM
I want to edit how your docs left-gutter nav works.....
requirements.txt + "//:name" syntax?
h

hundreds-father-404

10/01/2020, 5:27 PM
I want to edit how your docs left-gutter nav works.....
We use readme.io to host our docs. I don’t think we can edit this easily
w

witty-crayon-22786

10/01/2020, 5:27 PM
@billowy-motherboard-58443: no, the
module_mapping
aspect in particular: can find in page
b

billowy-motherboard-58443

10/01/2020, 5:29 PM
I don't think that's my problem ATM
the bit right after that with psycopg2 is more what I was looking for? 🙂
w

witty-crayon-22786

10/01/2020, 5:30 PM
@billowy-motherboard-58443: if the requirement name is
cassandra-driver
but the import is
cassandra
, then it’s likely that it will not be inferred
b

billowy-motherboard-58443

10/01/2020, 5:30 PM
I'm guessing the module_mapping is about avoiding collisions?
ahhhhh
the way the doc is written, it seems like it is intended to address collisions
w

witty-crayon-22786

10/01/2020, 5:31 PM
i’ll add an aside there
b

billowy-motherboard-58443

10/01/2020, 5:31 PM
is the intent
"published-name": "name-one-imports" ?
👍 1
w

witty-crayon-22786

10/01/2020, 5:32 PM
yep
too many names for each of those things i suppose 😃
i added one additional combination to the page
b

billowy-motherboard-58443

10/01/2020, 5:34 PM
ok, I used an explicit "//:cassandra-driver" in python_distribution(dependencies=)
which got it into the setup.py
but it ignored the version I had in constraints.txt
it had no version at all in setup.py
w

witty-crayon-22786

10/01/2020, 5:35 PM
it will be the version from your requirements.txt
but note also that putting the dependency in the distribution is the wrong place
if your code depends on that requirement, the code should have the dependency.
b

billowy-motherboard-58443

10/01/2020, 5:35 PM
better in the python_library() ?
w

witty-crayon-22786

10/01/2020, 5:36 PM
if you intend to use inference, would suggest fixing the mapping so that you don’t need to manually add the dep.
yes
b

billowy-motherboard-58443

10/01/2020, 5:36 PM
kk
whooop
as soon as I did that, it complained about no separate requirements.txt in that lib's subtree
I had the impression only 1 top-level requirements.txt was used....
w

witty-crayon-22786

10/01/2020, 5:39 PM
you can have as many as you like, but we recommend one for now, yea
the BUILD content (
python_requirements(..)
) should be next to the requirements.txt, wherever it is.
h

hundreds-father-404

10/01/2020, 5:40 PM
What’s the error message? (It’s helpful for us to always include the original error message for things. If it’s huge, Slack’s code snippet feature is helpful, or Gists/Pastebin)
b

billowy-motherboard-58443

10/01/2020, 5:40 PM
Copy code
MappingError: Failed to parse gh/leader_election/BUILD:
[Errno 2] No such file or directory: '/src/gh-pants-test/gh/leader_election/requirements.txt'
top of tree is gh-pants-test
👍 1
which is where requirements.txt and constraints.txt live
the
python_requirements(..)
definition
b

billowy-motherboard-58443

10/01/2020, 5:43 PM
so move the python_requirements() to the top level build, rather than the one with the python_lib and python_dist declarations for the particular pkg I'm trying to build?
👍 1
sort of got it, but still getting no version in the setup.py
(for cassandra-driver
it doesn't check constraints.txt for that?
h

hundreds-father-404

10/01/2020, 5:47 PM
For the file that imports
cassandra
, you can run
./pants dependencies path/to/file.py
to check that
//:cassandra-driver
is being properly inferred
b

billowy-motherboard-58443

10/01/2020, 5:48 PM
yes, it gets that
but my problem is this part of the generated setup.py
Copy code
'install_requires': (
        'cassandra-driver',
        'gh.config==0.1',
        'gh.core==0.1',
        'gh.util==0.1',
    ),
no version for cassandra-driver
where I do have it set in constraints.txt
(but no where else)
h

hundreds-father-404

10/01/2020, 5:49 PM
it doesn’t check constraints.txt for that?
Not for dependency inference. The
python_requirements()
target is a macro that reads your
requirements.txt
and creates a bunch of
python_requirement_library
targets for you automatically. One target per each distinct requirement. Dependency inference uses this when analyzing your imports. Meanwhile, the
constraints.txt
is solely used by Pex (which uses pip) when resolving requirements.
b

billowy-motherboard-58443

10/01/2020, 5:49 PM
kk
and yes, just proved to myself that that works
my top level BUILD is now
Copy code
python_requirements()

python_requirement_library(
    name="cassandra",
    requirements=["cassandra-driver==3.24"],
    module_mapping={
        "cassandra-driver": ["cassandra"],
    },
)
`
h

hundreds-father-404

10/01/2020, 5:50 PM
Oh, okay, thanks for including that snippet. If you want to constrain
cassandra-driver
in your generated
setup.py
, then you need the constraint that you want in
requirements.txt
We very intentionally do not use your constraints.txt when generating setup.py. You might want your setup.py to be loose, like
>=3.0,<4
, whereas a
constraints.txt
should be pinning to a specific version like
==3.3
b

billowy-motherboard-58443

10/01/2020, 5:50 PM
does that seem sensible?
oh, so it WILL use reruirements.txt in generating setup.py?
h

hundreds-father-404

10/01/2020, 5:51 PM
That’s fine to do it that way too, if you don’t want to have
cassandra-driver
in your
requirements.txt
. If it’s okay to be in your
requirements.txt
, that’s simpler than explicitly defining a
python_requirement_library
You would update your
requirmenets.txt
to be
cassandra-driver==3.24
, whereas right now it sounds like it’s
cassandra-driver
b

billowy-motherboard-58443

10/01/2020, 5:52 PM
hmm, seems not to
I just tried with taking the vesion out of of python_requirement_library and put it in requirements.txt
and it does NOT get included in setup.py
h

hundreds-father-404

10/01/2020, 5:53 PM
To summarize, I’d recommend: 1. move the
module_mapping
to your
python_requirements()
macro 2. delete the
python_requirement_library
target. This is important for dependency inference to work, otherwise you will have two targets describing the same requirement, and dep inference won’t know what to do with the ambiguity 3. Update your
requirements.txt
to use
==3.3
b

billowy-motherboard-58443

10/01/2020, 5:54 PM
3 didn't help - not reflected in the generated setup.py
I can try 1 &2
h

hundreds-father-404

10/01/2020, 5:54 PM
Did you first do #1 and #2?
b

billowy-motherboard-58443

10/01/2020, 5:55 PM
an no
ah, ok, worked
top level BUILD is now
Copy code
python_requirements(
    # mapping for "pip install name is different from import name"
    module_mapping={
        "cassandra-driver": ["cassandra"],
    },
)
💯 1
and that's it
h

hundreds-father-404

10/01/2020, 5:57 PM
I suspect what happened is that if you didn’t first do #1, then dependency inference will not end up using what’s defined in your
requriements.txt
. Pants doesn’t know that
import cassandra
means
cassandra-driver
. So updating
requirements.txt
doesn’t do anything. But, it does know about the inline target you created in the BUILD file, which is why the value is still showing up. -- Now, once you do #1 but not yet #2, you have the issue that you now have two distinct targets that are both mapped to the import
cassandra
. This results in neither target being used, and nothing showing up at all. Dependency inference cannot disambiguate what you want. -- It requires all 3 steps to work properly.
Awesome!
12 Views