Hey everyone, we are just getting started with Pants here and I'm trying to write a little python app that can generate things using protobufs. The kicker is I'm trying to do it generically which is causing me some issues with resolving dependencies in the build process. I created an example repo to help show what I'm seeing: https://github.com/fishst1k/pants-proto/tree/main TLDR is I want to use

importlib.import_module

to dynamically import different files. Since I'm not explicitly importing in python,

pants

doesn't do the dependency magic to find files and include them in the build sandbox. That's fine because I can update the

BUILD

to define the resource. But since the file I am importing is also using codegen protos it doesn't seem to find the

pb2

files and include them in the sandbox either! So my first question was can I import the

pb2

files? It doesn't appear that I can, but I'm new to this so maybe I am wrong.

File "/private/var/folders/jv/9rg1vfp12plc7px03gr7qryr0000gn/T/pants-sandbox-ooemt6/src/python/app/data/example/examples.py", line 1, in <module>
    from example.v1.example_pb2 import Example
ModuleNotFoundError: No module named 'example'

The work around I have for now is I can explicitly import my example data file and pants finds everything! But that kind of defeats the purpose of generically using

importlib.import_module

.

import importlib

from google.protobuf.json_format import MessageToJson

# Uncomment this to explicitly import the data file
# and the app will work!
# import app.data.example.examples


if __name__ == "__main__":
    libs = ["examples"]
    for name in libs:
        lib = importlib.import_module(f"app.data.example.{name}")

I dunno, if anyone has any ideas for me I'd love to hear them!

did you add those files as a dependency for that script?

the proto files are being imported by the file i am "generically" importing here if that's what you mean?

hmm, i wonder if you're missing a

BUILD

file and a

__init__.py

file here? https://github.com/fishst1k/pants-proto/tree/main/src/python/app/data

yeah, i was hoping it was that easy too...but when i tried it didn't seem work

oh actually i think that's the wrong path, i misread the error message: what about adding those files here: https://github.com/fishst1k/pants-proto/tree/main/src/protobuf/example? (also is the file supposed to be

example_pb2

or

example.proto

in that traceback?)

the

pb2

is the compiled proto in python form

hmm, i see. I'm not familiar with protobuf, so i think i'm outside my depth here. i'll let someone smarter than me step in 🙂

i really appreciate you helping...thank you!

You may have luck with https://www.pantsbuild.org/2.20/reference/subsystems/python-infer#string_imports, and not doing dynamic string formatting with your strings. So

libs = ["app.data.example.examples"]
    for lib_name in libs:
        lib = importlib.import_module(lib_name)

that's an interesting idea...i'll give it a shot here

Hmm, I was able to get it to work:

diff --git a/pants.toml b/pants.toml
index e9ba168..fd39a0f 100644
--- a/pants.toml
+++ b/pants.toml
@@ -27,3 +27,6 @@ mypy_plugin = true
 [python]
 interpreter_constraints = ["CPython>=3.8,<3.11"]
 enable_resolves = true
+
+[python-infer]
+string_imports = true
diff --git a/src/python/app/main.py b/src/python/app/main.py
index 764c4da..27d7661 100644
--- a/src/python/app/main.py
+++ b/src/python/app/main.py
@@ -8,9 +8,9 @@ from google.protobuf.json_format import MessageToJson
 
 
 if __name__ == "__main__":
-    libs = ["examples"]
-    for name in libs:
-        lib = importlib.import_module(f"app.data.example.{name}")
+    libs = {"app.data.example.examples": "examples"}
+    for path, name in libs.items():
+        lib = importlib.import_module(path)
 
         data = getattr(lib, name)
         for item in data:

~/workspace/pants-proto $ pants run src/python/app/main.py
12:37:43.61 [INFO] Initialization options changed: reinitializing scheduler...
12:37:46.47 [INFO] Scheduler initialized.
{
  "name": "example1"
}
{
  "name": "example2"
}

oh!

+[python-infer]
+string_imports = true

Ah yeah, it's not enabled by default!

indeed, that's awesome! thank you - i can definitely work with that