hi pants, dependency hell again :slightly_smiling_...
# general
a
hi pants, dependency hell again 🙂 I have put myself in a weird situation where I think Pants is not properly detecting internal dependencies because we do some dynamic loading, and I don’t know how to solve this. Let’s see if I can explain.
I have the following test
Copy code
import pytest

from namespace.notifications import get_notifier
from namespace.notifications import notifier


def test_double_definition():
    """Check protection against notifier duplication."""
    _ = get_notifier("slack")
    with pytest.raises(AttributeError):

        class DuplicatedNotifier(notifier.Notifier, name="slack"):  # NOQA
            """Duplicated notifier name."""
The key is in
get_notifier
, which is in `namespace/notifications/___init___.py`This function does black magic to detect configured notifiers in a sort of plugin-like architecture
Copy code
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""Load notification plugins."""

import pkgutil
import importlib


def get_notifier(name):
    """Load the correct notifier.

    Imports all modules in `proton.common.notifiers` and checks the loaded
    grabbers in Notifier.

    Parameters
    ----------
    name : str
        Name of the Notifier to load.

    Returns
    -------
    Notifier
        None if not found.

    """
    from namespace.notifications import notifiers as __notifiers
    from namespace.notifications.notifier import Notifier

    if name not in Notifier.NOTIFIERS:
        for _, module_name, _ispkg in pkgutil.iter_modules(
            __notifiers.__path__, __notifiers.__name__ + "."
        ):
            importlib.import_module(module_name)
    if name in Notifier.NOTIFIERS:
        return Notifier.NOTIFIERS[name]
    return None
The top part of
notifier.py
Copy code
class Notifier:
    """Generic notifier.

    Attributes
    ----------
    REQUIRED_KEYS : set
        See below.
    SERVICE_KEYS : set
        See below.
    NOTIFIERS : dict
        Autodiscovered subclasse, i.e., available notifiers.

    Parameters
    ----------
    environment : str
        Environment in which the app is running (local, qa, production...)
    common_config : dict
        Common config as specified in proton.common/config_data/notifiers.
    notifier_config : dict
        Notifier-specific config. Required entries in the dict are defined in
        REQUIRED_KEYS.
    service_config : dict
        Config of the notification service, with required entries defined in SERVICE_KEYS.
    """

    NOTIFIERS = {}
    REQUIRED_KEYS = set()
    SERVICE_KEYS = set()

    def __init_subclass__(cls, name, **kwargs):
        """Add grabber class to loaded grabbers list."""
        super().__init_subclass__(**kwargs)
        cls.name = name
        if name:
            if name in Notifier.NOTIFIERS:
                raise AttributeError(f"Notifier defined twice -> {name}")
            Notifier.NOTIFIERS[name] = cls
And then I’d have notifiers in
namespace/notifications/notifiers/*.py
. The problem is that unless I explicitly load some notifiers in my test above, it fails with
Copy code
>       from namespace.notifications import notifiers as __notifiers
E       ImportError: cannot import name 'notifiers' from 'namespace.notifications' (lib/notifications/namespace/notifications/__init__.py)
This works, for example:
Copy code
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""Test the get_notifier functionality."""

from namespace.notifications import get_notifier
from namespace.notifications.notifiers.slack import SlackNotifier


def test_get_slack_notifier():
    """Load Slack notifier."""
    notifier = get_notifier("slack")
    assert notifier == SlackNotifier
I think dependency inference is not working in this case because I need to be explicit and say: hey, I always want to include the
notifiers
folder, because it’s going to be loaded in weird ways. I have looked at the docs and I don’t see how to do it (I only see it with 3rd party deps). Any hints?
c
Does it work to add the required libraries as
dependencies
on your tests targets?
i.e. manually do what the dependency inference is failing to do for you here.
a
I would be adding itself? because it’s a sub-dependency
it’s a folder inside a dependency, so it’s a part of the library
if I go to the
repl
it loads normally and I don’t get the error
c
You can try
./pants dependencies --transitive test/target
to see what it pulls in, and compare with your distribution, or library or what you have.
I’ve been caught off guard once, where I had code in the tree, along with my other library code, and some of it were left out, due to no one explicitly importing it.
a
yeah
this is correct
this is what’s happening
in some tests it works, in some others it’s not
because of the imports
in some tests there is an explicit import, in some others there’s not
but it’s always implicitly done
so if each test is an independent pex, it seems like some pex pick up the dependency and some others don’t
c
I think figuring out what you need to add to your dependencies to make it explicit will solve this for you.
Can you share the contents of a relevant BUILD file?
a
yup,
macros.py
Copy code
def tests(**tests_kwargs):
    # Setup tests
    python_tests(
        name="unit",
        sources=["tests/unit/*.py", "tests/*.py"],
        **tests_kwargs,
    )
    python_tests(
        name="integration",
        sources=["tests/integration/*.py"],
        tags=["integration"],
        **tests_kwargs,
    )


def lib(
    name,
    has_config=True,
    module_mapping=None,
    lib_kwargs=None,
    tests_kwargs=None,
    req_kwargs=None,
):
    # Create the library
    lib_kwargs = {} if not lib_kwargs else lib_kwargs
    if has_config:
        resources(name="config", sources=[f"namespace/{name}/config_data/*"])
        if not "dependencies" in lib_kwargs:
            lib_kwargs["dependencies"] = []
        lib_kwargs["dependencies"].append(":config")
    python_library(name=name, sources=[f"namespace/{name}/**/*.py"], **lib_kwargs)

    # Setup tests
    tests_kwargs = {} if not tests_kwargs else tests_kwargs
    tests()

    # Requirements
    req_kwargs = {} if not req_kwargs else req_kwargs
    if not module_mapping:
        module_mapping = {}
    python_requirements(module_mapping=module_mapping, **req_kwargs)
and
BUILD
file
Copy code
lib("notifications",
    has_config=True,
    module_mapping={"slackclient": ["slack"]})
dir structure
Copy code
lib/notifications
├── BUILD
├── README.md
├── namespace
│   └── notifications
│       ├── __init__.py
│       ├── center.py
│       ├── config_data
│       │   └── notifiers.yaml
│       ├── levels.py
│       ├── notifier.py
│       └── notifiers
│           ├── email.py
│           └── slack.py
├── requirements.txt
└── tests
    ├── test_get_notifier.py
    ├── test_levels.py
    ├── test_notification_center.py
    ├── test_notifications.py
    ├── test_notifier.py
    ├── test_notifiers_email.py
    └── test_notifiers_slack.py
c
So, what does
./pants dependencies lib/notifications:notifications
and
./pants dependencies lib/notifications:unit
say?
a
Copy code
$ ./pants dependencies lib/notifications:notifications
lib/common/namespace/common/config.py:../../common
lib/common/namespace/common/utils.py:../../common
lib/notifications/namespace/notifications/__init__.py:../../notifications
lib/notifications/namespace/notifications/center.py:../../notifications
lib/notifications/namespace/notifications/config_data/notifiers.yaml:../../../config
lib/notifications/namespace/notifications/levels.py:../../notifications
lib/notifications/namespace/notifications/notifier.py:../../notifications
lib/notifications/namespace/notifications/notifiers/email.py:../../../notifications
lib/notifications/namespace/notifications/notifiers/slack.py:../../../notifications
lib/notifications:slackclient
Copy code
$ ./pants dependencies lib/notifications:unit
16:02:30.30 [WARN] Unmatched glob from lib/notifications:unit's `sources` field: "lib/notifications/tests/unit/*.py"

Do the file(s) exist? If so, check if the file(s) are in your `.gitignore` or the global `pants_ignore` option, which may result in Pants not being able to see the file(s) even though they exist on disk. Refer to <https://www.pantsbuild.org/v2.6/docs/troubleshooting#pants-cannot-find-a-file-in-your-project>.
//:pytest
lib/notifications/namespace/notifications/__init__.py:../../notifications
lib/notifications/namespace/notifications/center.py:../../notifications
lib/notifications/namespace/notifications/levels.py:../../notifications
lib/notifications/namespace/notifications/notifier.py:../../notifications
lib/notifications/namespace/notifications/notifiers/email.py:../../../notifications
lib/notifications/namespace/notifications/notifiers/slack.py:../../../notifications
lib/notifications/tests/test_get_notifier.py:../unit
lib/notifications/tests/test_levels.py:../unit
lib/notifications/tests/test_notification_center.py:../unit
lib/notifications/tests/test_notifications.py:../unit
lib/notifications/tests/test_notifier.py:../unit
lib/notifications/tests/test_notifiers_email.py:../unit
lib/notifications/tests/test_notifiers_slack.py:../unit
c
Hmm, yeah, OK, that actually seems reasonably right, to me, so I’d have to defer to a more experienced Pants user/maintainer to chime in here 😉 Perhaps @hundreds-father-404 or @happy-kitchen-89482 will spot on a glimpse what’s a miss here.. ?
e
namespace/___init__.py_
is missing
@astonishing-london-2419 in your tree output that stands out. You probably don't want to try mixing explicit packages and implicit namespace packages. Is that being done with intent?
c
namespace/notifiers doesn’t seem to have a
___init___.py
file either..
a
the init in namespace is intentional, because it’s a namespace package
but notifiers is a mistake
now tests fail in different ways, let me investigate
e
So, do you support Python2.7 and thus can't use implicit namespace packages? I'm a bit confused.
a
no no, it’s only python 3 so we use implicit namespace packages
but there’s the init because we had some code in there
e
the init in namespace is intentional, because it’s a namespace package
This is confusing me then. There is no need for an
__init__.py
there to create a namespace package.
a
the init in namespace is intentional, because it’s a namespace package
I’m a moron: “the lack of init”
e
Gotcha.
a
(also note, this is a migration of an old repo and all tests passed with nox, that’s why I’m asking so much)
h
Pants is very strict about sandboxing, so sometimes tests happen to pass outside Pants due to dependency leakage but will fail in Pants. Not sure if that is what's happening here though.
a
it looks like there’s some issue with Sandboxing, yes… the plugins don’t seem to load properly
h
but there’s the init because we had some code in there
Ah ha. See the tool tip in https://www.pantsbuild.org/docs/python-backend about init.py files. Common gotcha that we want to improve https://github.com/pantsbuild/pants/issues/10823
e
@astonishing-london-2419 I think the recursive glob in the macro is shooting you in the foot. You'll need to have a target just for the notifiers and then add an explicit dependency on that target.
a
Ah ha. See the tool tip in https://www.pantsbuild.org/docs/python-backend about init.py files
yup, i have
infer=True
👍 1
@astonishing-london-2419 I think the recursive glob in the macro is shooting you in the foot. You’ll need to have a target just for the notifiers and then add an explicit dependency on that target.
interesting, how so? (I can totally try, but I wanted to understand why)
e
For the sake of simplifying debugging, I'd start by not using the macro in this case.
Just define a normal target by hand that globs everything except the notifiers, then add a target to own the notifiers, then add that notifiers target to the 1st target's explicit dependencies.
a
yup, it seems to be something like that, because
pkgutil
is not finding the notifiers either
e
If that works, then considering how to macrofy / etc is just details and opinions. At least we'd have the mechanism of the issue sorted.
👍 1
a
I did the following in
BUILD
Copy code
resources(name="config", sources=["namespace/notifications/config_data/*"])

python_tests(
    name="unit",
    sources=["tests/unit/*.py", "tests/*.py"],
)
python_tests(
    name="integration",
    sources=["tests/integration/*.py"],
    tags=["integration"],
)

python_library(name="notifications", 
               sources=["namespace/notifications/**/*.py", 'namespace/notifications/notifiers/*.py'], 
               dependencies=[":config"])

python_requirements(module_mapping={"slackclient": ["slack"]})
and tree is
Copy code
$ tree lib/notifications
lib/notifications
├── BUILD
├── README.md
├── namespace
│   └── notifications
│       ├── __init__.py
│       ├── center.py
│       ├── config_data
│       │   └── notifiers.yaml
│       ├── levels.py
│       ├── notifier.py
│       └── notifiers
│           ├── __init__.py
│           ├── email.py
│           └── slack.py
├── pyproject.toml
├── requirements.txt
└── tests
    ├── test_get_notifier.py
    ├── test_levels.py
    ├── test_notification_center.py
    ├── test_notifications.py
    ├── test_notifier.py
    ├── test_notifiers_email.py
    └── test_notifiers_slack.py
still fails
e
It should. Just a sec for what I meant ...
a
pytest-mock
can’t find the modules, but something is fishy because the plugins are not getting loaded unless I explicitly load them
e
This is what I meant for you to try:
Copy code
resources(
    name="config",
    sources=["namespace/notifications/config_data/*"],
)

python_tests(
    name="unit",
    sources=["tests/unit/*.py", "tests/*.py"],
)
python_tests(
    name="integration",
    sources=["tests/integration/*.py"],
    tags=["integration"],
)

python_library(
    name="notifiers", 
    sources=["namespace/notifications/notifiers/*.py"],
)

python_library(
    name="notifications", 
    sources=["namespace/notifications/**/*.py", "!namespace/notifications/notifiers/*.py"], 
    dependencies=[
      ":config",
      ":notifiers",
    ],
)

python_requirements(
    module_mapping={
      "slackclient": ["slack"],
    },
)
Note the
notifications
libraray now excludes the notifiers, which get their own target. It then adds a manual dependency on that target.
a
oh, I see
I will try this, thanks!
e
I messed up the syntax I think, the
!
goes inside the string.
Ok, edited.
a
yup, I saw this
it works, thanks~
can I ask why? I am not sure I understand why this works and it didn’t before
this library basically does what I wanted to do (create an explicit dependency) but of course I don’t understand why it didn’t work as it was
e
I'll have to defer to @hundreds-father-404. Our internal representation of targets and external representation are different and my fix was based on a handwavy idea of what was going on. I'm pretty sure Eric will be able to say - "It;s this interaction between file deps / subtargets and dependency inference..." - with authority.
🙏 1
h
Our internal representation of targets and external representation are different and my fix was based on a handwavy idea of what was going on.
Yeah targets are really confusing right now..In fact, my main focus for this week has been a proposal at https://docs.google.com/document/d/1HpJn2jTWf5sKob6zhe4SqHZ7KWBlX4a6OJOwnuw-IHo/edit# to clean this all up So, right now, when you say
python_library(sources=["*.py"])
, Pants is really generating one target per source file. They each have their own dependencies, which you can see by running
./pants dependencies namespace/notifications/foo.py
vs.
./pants dependencies namespace/notifications/bar.py
. Those dependencies will be everything that you explicitly put in the
BUILD
file, plus whatever was inferred for that specific file. It's confusing that Pants is not transparent with this target generation, hence the proposal. The generated file-level targets do not automatically depend on the other generated targets from the
python_library
. That means that
notifications/foo.py
does not depend on
notifications/bar.py
. They will only depend on sibling files if dependency inference detects the import, or you explicitly add it. That's what you're doing by having two distinct
python_library
targets here: by putting
:notifications
in the
dependencies
field, you say that every generated target from the
notifications
target no matter what should depend on everything in the
:notifications
python_library
target. This is really hard to reason about, and not at all transparent. Does that sort of make sense? Feedback appreciated on the proposal to fix this all!
a
understood
I think this is what I intuitively thought was happening (since there is a bit of dynamic loading so the inference doesn’t work), but this makes it much clearer
❤️ 1
thanks so much for the help!
w
The generated file-level targets do not automatically depend on the other generated targets from the 
python_library
. That means that 
notifications/foo.py
 does not depend on 
notifications/bar.py
. They will only depend on sibling files if dependency inference detects the import, or you explicitly add it. That’s what you’re doing by having two distinct 
python_library
 targets here: by putting 
:notifications
 in the 
dependencies
 field, you say that every generated target from the 
notifications
 target no matter what should depend on everything in the 
:notifications
 
python_library
 target.
this is all true. but note that to support this case, we allow you to have a target depend on one of the files in the target… or on itself. which has the result that every file in the target depends on that file in the target (or on the entire target)
so rather than splitting the target into
notifications
and
notifiers
, you could also:
Copy code
python_library(
    name="notifications", 
    sources=["namespace/notifications/**/*.py"], 
    dependencies=[
      ":config",
      # This target depends on itself because dependencies on the
      # `notifiers` package are not inferred, and must be included manually.
      ":notifications",
    ],
)
👀 1
h
@witty-crayon-22786 I don't think that's right, that only works with a "file address" - pretty sure what you put results in a cycle
w
i just tried it and it worked.
👀 1
so that’s something to investigate independently!
e
Interesting. That seemed too radical to work to me when I suggested the seperate target fix.
But we do allow cycles for this sort of target so makes sense I guess.
h
I think here this works because we "expand"
:notifications
into all of its generated file-level targets. We allow cycles with file-level targets, so it's legal. Tricky!
a
is one or the other preferred? somehow this second one is more “self-contained” but I am not sure if it could have downsides
w
the primary downside is that it is less specific: the
notifiers
don’t need to be forced to depend on one another, but that’s the effect here (in addition to all of the other files). but unless we’re talking about many frequently changing files and binaries, i wouldn’t worry about it.
🙏 1