question regarding using

importlib

and pants detecting dependencies: I’ve got a class which dynamically imports another class using importlib:

# importer.py
import importlib
import classes

class Importer:
    def __init__(self, name):
        source_module = importlib.import_module(f"{classes.__package__}.s_{name}")
        class_name = inflection.camelize(name)
        self.source_parser = getattr(source_module, f"Class{class_name}")

here’s a simplified view of what the code looks like

project
|   BUILD
|   __init__.py
│   importer.py   
│
└───classes
|   |   BUILD
│   │   __init__.py
|   |   s_bar.py
|   |   s_baz.py
│   │   s_foo.py
|   |   ...

the issue is that pants can’t accurately infer that all of the modules under

classes/

are needed by the

Importer

class. It is obviously able to do so correctly if i explicitly import

s_bar, s_baz, s_foo...

, but there are a lot of classes and that feels redundant. I’m already using

classes.__package__

instead of simply using a hardcoded path string in the hopes that it would resolve the issue, but it looks like it only imports

classes.__init__

. 😞 what’s the right way to solve this?

Hello! One way to handle this is to explicitly add the dependencies field. However if you find that is getting to tedious and you are using this pattern a lot, it might be worth adding a dependency inference plug-in implementation. You can augment pant's dependency inference it is already using. wdyt?

hmm, i hadn’t remembered to explicitly add the classes as a dependency in the build file, and that should work, of course! one potential drawback is that this would make any sibling files of

importer.py

also have

classes/

as a dependency, which isn’t correct. ideally only

importer.py

would be dependent on

classes/

, but if there’s no easy solution then this is perfectly suitable. we use this pattern a little but probably not enough to build a plugin for. good to know that the option exists though. 🙂 p.s.: i hope you used speak-to-text instead of typing those messages out, people should try to rest when recovering from a fracture 😛

We strongly encourage using the overrides field for that exact reason, to have precise metadata. Precise metadata is really helpful to have finer grained caching. It also makes your build file easier to understand what is happening. https://www.pantsbuild.org/docs/targets#target-generation

Hey, it knows about Pants!

ah, perfect, that sounds like the solution i was looking for! so essentially this, right?

python_sources(
    overrides={
        "importer.py": {
            "dependencies": ["./classes"],
        },
    },
)

Something like that, not sure about the

./

prefix, but that's the idea