https://pantsbuild.org/ logo
Join Slack
Powered by
Hi, I have a question about imports.
# general
a
Hi, I have a question about imports.
☝️ 1
We are building a monorepo that will house our machine learning models. Broadly speaking, each model is contained in a directory that has a BUILD file. The setup looks something like this, where in this example there are 2 models: model_1 and model_2:
Copy code
monorepo/
	cicd/
		BUILD
		...
	libraries/
		BUILD
		...
	models/
		model_1/
			BUILD
			main.py
			model.py

		model_2/
			BUILD
			main.py
			model.py
	...
	pants
	pants.toml
The code in 
main.py
is identical across 
model_1
and 
model_2
. We started with 
model_1
and everything ran fine, but when we added model_2 and ran 
./pants check ::
we obtained this warning:
Copy code
The target models/model_1/main.py:lib imports `<http://model.bm|model.bm>`, but Pants cannot safely infer a dependency because more than one target owns this module, so it is ambiguous which to use: ['models/model_2/model.py:lib', 'models/model_1/model.py:lib'].

Please explicitly include the dependency you want in the `dependencies` field of models/model_1/main.py:lib, or ignore the ones you do not want by prefixing with `!` or `!!` so that one or no targets are left.

Alternatively, you can remove the ambiguity by deleting/changing some of the targets so that only 1 target owns this module. Refer to <https://www.pantsbuild.org/v2.12/docs/troubleshooting#import-errors-and-missing-dependencies>.
we were able to remove the warning by changing the imports to relative imports, eg from 
from model import bm
to 
from .model import bm
.
There was also an error:
Copy code
models/model_1/main.py: error: Duplicate module named "main" (also at "models/model_2/main.py")
models/model_1/main.py: note: Are you missing an __init__.py? Alternatively, consider using --exclude to avoid checking one of them
that we addressed by adding 
__init__.py
scripts to model_1 and model_2.
Is this the correct way to handle these imports and dependencies between modules?
b
Usually yeah. 
mypy
doesn't know you aren't using implicit namespace packages so it warns.
r
But that looks like a pants error, not a mypy error?
What makes this problem worse for us is the fact that we’re deploying these individual models (e.g. 
models/model_1
using MLFlow Project, which does not seem to handle relative imports. This means that we cannot use relative imports in MLFlow Projects, where the code actually runs, and we can’t use implied imports in Pants. Does that means we just can’t use MLFlow Project and Pants?
b
Whats your source roots?
r
You mean as defined in 
BUILD
?
b
in 
pants.toml
r
We’re using `marker_filenames`M for our
Copy code
marker_filenames = ["setup.cfg", "MLProject"]
setup.cfg
to identify our libraries 
MLProject
to identify our MLFlow Projects
b
Is this Pants v2?
a
2.12
b
I suspect you might wanna bump your source roots a dir higher. Then you can import as 
from model_1.model ...
☝️ 1
Open in Slack
PreviousNext
Powered by