astonishing-london-2419
09/14/2021, 1:16 PMastonishing-london-2419
09/14/2021, 1:16 PMimport pytest
from namespace.notifications import get_notifier
from namespace.notifications import notifier
def test_double_definition():
"""Check protection against notifier duplication."""
_ = get_notifier("slack")
with pytest.raises(AttributeError):
class DuplicatedNotifier(notifier.Notifier, name="slack"): # NOQA
"""Duplicated notifier name."""
The key is in get_notifier
, which is in `namespace/notifications/___init___.py`This function does black magic to detect configured notifiers in a sort of plugin-like architecture
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""Load notification plugins."""
import pkgutil
import importlib
def get_notifier(name):
"""Load the correct notifier.
Imports all modules in `proton.common.notifiers` and checks the loaded
grabbers in Notifier.
Parameters
----------
name : str
Name of the Notifier to load.
Returns
-------
Notifier
None if not found.
"""
from namespace.notifications import notifiers as __notifiers
from namespace.notifications.notifier import Notifier
if name not in Notifier.NOTIFIERS:
for _, module_name, _ispkg in pkgutil.iter_modules(
__notifiers.__path__, __notifiers.__name__ + "."
):
importlib.import_module(module_name)
if name in Notifier.NOTIFIERS:
return Notifier.NOTIFIERS[name]
return None
The top part of notifier.py
class Notifier:
"""Generic notifier.
Attributes
----------
REQUIRED_KEYS : set
See below.
SERVICE_KEYS : set
See below.
NOTIFIERS : dict
Autodiscovered subclasse, i.e., available notifiers.
Parameters
----------
environment : str
Environment in which the app is running (local, qa, production...)
common_config : dict
Common config as specified in proton.common/config_data/notifiers.
notifier_config : dict
Notifier-specific config. Required entries in the dict are defined in
REQUIRED_KEYS.
service_config : dict
Config of the notification service, with required entries defined in SERVICE_KEYS.
"""
NOTIFIERS = {}
REQUIRED_KEYS = set()
SERVICE_KEYS = set()
def __init_subclass__(cls, name, **kwargs):
"""Add grabber class to loaded grabbers list."""
super().__init_subclass__(**kwargs)
cls.name = name
if name:
if name in Notifier.NOTIFIERS:
raise AttributeError(f"Notifier defined twice -> {name}")
Notifier.NOTIFIERS[name] = cls
And then I’d have notifiers in namespace/notifications/notifiers/*.py
. The problem is that unless I explicitly load some notifiers in my test above, it fails with
> from namespace.notifications import notifiers as __notifiers
E ImportError: cannot import name 'notifiers' from 'namespace.notifications' (lib/notifications/namespace/notifications/__init__.py)
This works, for example:
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""Test the get_notifier functionality."""
from namespace.notifications import get_notifier
from namespace.notifications.notifiers.slack import SlackNotifier
def test_get_slack_notifier():
"""Load Slack notifier."""
notifier = get_notifier("slack")
assert notifier == SlackNotifier
I think dependency inference is not working in this case because I need to be explicit and say: hey, I always want to include the notifiers
folder, because it’s going to be loaded in weird ways. I have looked at the docs and I don’t see how to do it (I only see it with 3rd party deps). Any hints?curved-television-6568
09/14/2021, 1:18 PMdependencies
on your tests targets?curved-television-6568
09/14/2021, 1:19 PMastonishing-london-2419
09/14/2021, 1:19 PMastonishing-london-2419
09/14/2021, 1:19 PMastonishing-london-2419
09/14/2021, 1:20 PMrepl
it loads normally and I don’t get the errorcurved-television-6568
09/14/2021, 1:24 PM./pants dependencies --transitive test/target
to see what it pulls in, and compare with your distribution, or library or what you have.curved-television-6568
09/14/2021, 1:24 PMastonishing-london-2419
09/14/2021, 1:29 PMastonishing-london-2419
09/14/2021, 1:29 PMastonishing-london-2419
09/14/2021, 1:29 PMastonishing-london-2419
09/14/2021, 1:29 PMastonishing-london-2419
09/14/2021, 1:29 PMastonishing-london-2419
09/14/2021, 1:29 PMastonishing-london-2419
09/14/2021, 1:30 PMastonishing-london-2419
09/14/2021, 1:30 PMcurved-television-6568
09/14/2021, 1:32 PMcurved-television-6568
09/14/2021, 1:32 PMastonishing-london-2419
09/14/2021, 1:35 PMmacros.py
def tests(**tests_kwargs):
# Setup tests
python_tests(
name="unit",
sources=["tests/unit/*.py", "tests/*.py"],
**tests_kwargs,
)
python_tests(
name="integration",
sources=["tests/integration/*.py"],
tags=["integration"],
**tests_kwargs,
)
def lib(
name,
has_config=True,
module_mapping=None,
lib_kwargs=None,
tests_kwargs=None,
req_kwargs=None,
):
# Create the library
lib_kwargs = {} if not lib_kwargs else lib_kwargs
if has_config:
resources(name="config", sources=[f"namespace/{name}/config_data/*"])
if not "dependencies" in lib_kwargs:
lib_kwargs["dependencies"] = []
lib_kwargs["dependencies"].append(":config")
python_library(name=name, sources=[f"namespace/{name}/**/*.py"], **lib_kwargs)
# Setup tests
tests_kwargs = {} if not tests_kwargs else tests_kwargs
tests()
# Requirements
req_kwargs = {} if not req_kwargs else req_kwargs
if not module_mapping:
module_mapping = {}
python_requirements(module_mapping=module_mapping, **req_kwargs)
and BUILD
file
lib("notifications",
has_config=True,
module_mapping={"slackclient": ["slack"]})
astonishing-london-2419
09/14/2021, 1:35 PMlib/notifications
├── BUILD
├── README.md
├── namespace
│ └── notifications
│ ├── __init__.py
│ ├── center.py
│ ├── config_data
│ │ └── notifiers.yaml
│ ├── levels.py
│ ├── notifier.py
│ └── notifiers
│ ├── email.py
│ └── slack.py
├── requirements.txt
└── tests
├── test_get_notifier.py
├── test_levels.py
├── test_notification_center.py
├── test_notifications.py
├── test_notifier.py
├── test_notifiers_email.py
└── test_notifiers_slack.py
curved-television-6568
09/14/2021, 1:57 PM./pants dependencies lib/notifications:notifications
and ./pants dependencies lib/notifications:unit
say?astonishing-london-2419
09/14/2021, 2:03 PM$ ./pants dependencies lib/notifications:notifications
lib/common/namespace/common/config.py:../../common
lib/common/namespace/common/utils.py:../../common
lib/notifications/namespace/notifications/__init__.py:../../notifications
lib/notifications/namespace/notifications/center.py:../../notifications
lib/notifications/namespace/notifications/config_data/notifiers.yaml:../../../config
lib/notifications/namespace/notifications/levels.py:../../notifications
lib/notifications/namespace/notifications/notifier.py:../../notifications
lib/notifications/namespace/notifications/notifiers/email.py:../../../notifications
lib/notifications/namespace/notifications/notifiers/slack.py:../../../notifications
lib/notifications:slackclient
astonishing-london-2419
09/14/2021, 2:03 PM$ ./pants dependencies lib/notifications:unit
16:02:30.30 [WARN] Unmatched glob from lib/notifications:unit's `sources` field: "lib/notifications/tests/unit/*.py"
Do the file(s) exist? If so, check if the file(s) are in your `.gitignore` or the global `pants_ignore` option, which may result in Pants not being able to see the file(s) even though they exist on disk. Refer to <https://www.pantsbuild.org/v2.6/docs/troubleshooting#pants-cannot-find-a-file-in-your-project>.
//:pytest
lib/notifications/namespace/notifications/__init__.py:../../notifications
lib/notifications/namespace/notifications/center.py:../../notifications
lib/notifications/namespace/notifications/levels.py:../../notifications
lib/notifications/namespace/notifications/notifier.py:../../notifications
lib/notifications/namespace/notifications/notifiers/email.py:../../../notifications
lib/notifications/namespace/notifications/notifiers/slack.py:../../../notifications
lib/notifications/tests/test_get_notifier.py:../unit
lib/notifications/tests/test_levels.py:../unit
lib/notifications/tests/test_notification_center.py:../unit
lib/notifications/tests/test_notifications.py:../unit
lib/notifications/tests/test_notifier.py:../unit
lib/notifications/tests/test_notifiers_email.py:../unit
lib/notifications/tests/test_notifiers_slack.py:../unit
curved-television-6568
09/14/2021, 2:09 PMenough-analyst-54434
09/14/2021, 2:41 PMnamespace/___init__.py_
is missingenough-analyst-54434
09/14/2021, 2:42 PMcurved-television-6568
09/14/2021, 2:44 PM___init___.py
file either..astonishing-london-2419
09/14/2021, 3:00 PMastonishing-london-2419
09/14/2021, 3:00 PMastonishing-london-2419
09/14/2021, 3:01 PMenough-analyst-54434
09/14/2021, 3:01 PMastonishing-london-2419
09/14/2021, 3:01 PMastonishing-london-2419
09/14/2021, 3:02 PMenough-analyst-54434
09/14/2021, 3:03 PMthe init in namespace is intentional, because it’s a namespace packageThis is confusing me then. There is no need for an
__init__.py
there to create a namespace package.astonishing-london-2419
09/14/2021, 3:03 PMthe init in namespace is intentional, because it’s a namespace packageI’m a moron: “the lack of init”
enough-analyst-54434
09/14/2021, 3:04 PMastonishing-london-2419
09/14/2021, 3:05 PMhappy-kitchen-89482
09/14/2021, 3:14 PMastonishing-london-2419
09/14/2021, 3:15 PMhundreds-father-404
09/14/2021, 3:15 PMbut there’s the init because we had some code in thereAh ha. See the tool tip in https://www.pantsbuild.org/docs/python-backend about init.py files. Common gotcha that we want to improve https://github.com/pantsbuild/pants/issues/10823
enough-analyst-54434
09/14/2021, 3:15 PMastonishing-london-2419
09/14/2021, 3:16 PMAh ha. See the tool tip in https://www.pantsbuild.org/docs/python-backend about init.py filesyup, i have
infer=True
astonishing-london-2419
09/14/2021, 3:17 PM@astonishing-london-2419 I think the recursive glob in the macro is shooting you in the foot. You’ll need to have a target just for the notifiers and then add an explicit dependency on that target.interesting, how so? (I can totally try, but I wanted to understand why)
enough-analyst-54434
09/14/2021, 3:17 PMenough-analyst-54434
09/14/2021, 3:18 PMastonishing-london-2419
09/14/2021, 3:18 PMpkgutil
is not finding the notifiers eitherenough-analyst-54434
09/14/2021, 3:19 PMastonishing-london-2419
09/14/2021, 7:05 PMBUILD
resources(name="config", sources=["namespace/notifications/config_data/*"])
python_tests(
name="unit",
sources=["tests/unit/*.py", "tests/*.py"],
)
python_tests(
name="integration",
sources=["tests/integration/*.py"],
tags=["integration"],
)
python_library(name="notifications",
sources=["namespace/notifications/**/*.py", 'namespace/notifications/notifiers/*.py'],
dependencies=[":config"])
python_requirements(module_mapping={"slackclient": ["slack"]})
and tree is
$ tree lib/notifications
lib/notifications
├── BUILD
├── README.md
├── namespace
│ └── notifications
│ ├── __init__.py
│ ├── center.py
│ ├── config_data
│ │ └── notifiers.yaml
│ ├── levels.py
│ ├── notifier.py
│ └── notifiers
│ ├── __init__.py
│ ├── email.py
│ └── slack.py
├── pyproject.toml
├── requirements.txt
└── tests
├── test_get_notifier.py
├── test_levels.py
├── test_notification_center.py
├── test_notifications.py
├── test_notifier.py
├── test_notifiers_email.py
└── test_notifiers_slack.py
astonishing-london-2419
09/14/2021, 7:05 PMenough-analyst-54434
09/14/2021, 7:06 PMastonishing-london-2419
09/14/2021, 7:06 PMpytest-mock
can’t find the modules, but something is fishy because the plugins are not getting loaded unless I explicitly load themenough-analyst-54434
09/14/2021, 7:10 PMresources(
name="config",
sources=["namespace/notifications/config_data/*"],
)
python_tests(
name="unit",
sources=["tests/unit/*.py", "tests/*.py"],
)
python_tests(
name="integration",
sources=["tests/integration/*.py"],
tags=["integration"],
)
python_library(
name="notifiers",
sources=["namespace/notifications/notifiers/*.py"],
)
python_library(
name="notifications",
sources=["namespace/notifications/**/*.py", "!namespace/notifications/notifiers/*.py"],
dependencies=[
":config",
":notifiers",
],
)
python_requirements(
module_mapping={
"slackclient": ["slack"],
},
)
enough-analyst-54434
09/14/2021, 7:10 PMnotifications
libraray now excludes the notifiers, which get their own target. It then adds a manual dependency on that target.astonishing-london-2419
09/14/2021, 7:14 PMastonishing-london-2419
09/14/2021, 7:14 PMenough-analyst-54434
09/14/2021, 7:15 PM!
goes inside the string.enough-analyst-54434
09/14/2021, 7:15 PMastonishing-london-2419
09/14/2021, 7:16 PMastonishing-london-2419
09/14/2021, 7:16 PMastonishing-london-2419
09/14/2021, 7:16 PMastonishing-london-2419
09/14/2021, 7:17 PMenough-analyst-54434
09/14/2021, 7:24 PMhundreds-father-404
09/14/2021, 7:59 PMOur internal representation of targets and external representation are different and my fix was based on a handwavy idea of what was going on.Yeah targets are really confusing right now..In fact, my main focus for this week has been a proposal at https://docs.google.com/document/d/1HpJn2jTWf5sKob6zhe4SqHZ7KWBlX4a6OJOwnuw-IHo/edit# to clean this all up So, right now, when you say
python_library(sources=["*.py"])
, Pants is really generating one target per source file. They each have their own dependencies, which you can see by running ./pants dependencies namespace/notifications/foo.py
vs. ./pants dependencies namespace/notifications/bar.py
. Those dependencies will be everything that you explicitly put in the BUILD
file, plus whatever was inferred for that specific file. It's confusing that Pants is not transparent with this target generation, hence the proposal.
The generated file-level targets do not automatically depend on the other generated targets from the python_library
. That means that notifications/foo.py
does not depend on notifications/bar.py
. They will only depend on sibling files if dependency inference detects the import, or you explicitly add it. That's what you're doing by having two distinct python_library
targets here: by putting :notifications
in the dependencies
field, you say that every generated target from the notifications
target no matter what should depend on everything in the :notifications
python_library
target.
This is really hard to reason about, and not at all transparent. Does that sort of make sense? Feedback appreciated on the proposal to fix this all!astonishing-london-2419
09/14/2021, 8:04 PMastonishing-london-2419
09/14/2021, 8:05 PMastonishing-london-2419
09/14/2021, 8:05 PMwitty-crayon-22786
09/14/2021, 8:16 PMThe generated file-level targets do not automatically depend on the other generated targets from thethis is all true. but note that to support this case, we allow you to have a target depend on one of the files in the target… or on itself. which has the result that every file in the target depends on that file in the target (or on the entire target). That means thatpython_library
does not depend onnotifications/foo.py
. They will only depend on sibling files if dependency inference detects the import, or you explicitly add it. That’s what you’re doing by having two distinctnotifications/bar.py
targets here: by puttingpython_library
in the:notifications
field, you say that every generated target from thedependencies
target no matter what should depend on everything in thenotifications
:notifications
target.python_library
witty-crayon-22786
09/14/2021, 8:17 PMnotifications
and notifiers
, you could also:
python_library(
name="notifications",
sources=["namespace/notifications/**/*.py"],
dependencies=[
":config",
# This target depends on itself because dependencies on the
# `notifiers` package are not inferred, and must be included manually.
":notifications",
],
)
hundreds-father-404
09/14/2021, 8:18 PMwitty-crayon-22786
09/14/2021, 8:19 PMwitty-crayon-22786
09/14/2021, 8:20 PMenough-analyst-54434
09/14/2021, 8:21 PMenough-analyst-54434
09/14/2021, 8:22 PMhundreds-father-404
09/14/2021, 8:23 PM:notifications
into all of its generated file-level targets. We allow cycles with file-level targets, so it's legal. Tricky!astonishing-london-2419
09/14/2021, 8:24 PMwitty-crayon-22786
09/14/2021, 8:27 PMnotifiers
don’t need to be forced to depend on one another, but that’s the effect here (in addition to all of the other files). but unless we’re talking about many frequently changing files and binaries, i wouldn’t worry about it.