https://pantsbuild.org/ logo
Join Slack
Powered by
Hello everyone, I would appreciate some help on 3r...
# general
w
Hello everyone, I would appreciate some help on 3rd party owner inference for a python based monorepo with 2 resolves. the problem I am facing is with confluent-kafka being imported as confluent_kafka.
My resolves are defines like this:
Copy code
python_requirements(
    source="requirements/default.txt",
    module_mapping={
        "confluent_kafka": ["confluent-kafka"],
        "confluent_kafka.schema_registry": ["confluent-kafka"],
        "confluent_kafka.schema_registry.avro": ["confluent-kafka"],
        "confluent_kafka.schema_registry.json_schema": ["confluent-kafka"],
        "confluent_kafka.schema_registry.serde": ["confluent-kafka"],
        "discord": ["discord.py"],
    },
)

python_requirements(
    name="pants-plugins",
    resolve="pants-plugins",
    source="requirements/pants.txt",
)
my source is defined like this (using a macro:
Copy code
python_sources(
        name="sources",
        description=description,
        sources=[
            f"{name}/**/*.py",
            f"!{name}/resources/**/*.py",
            f"!{name}/schemas/**/*.py",
            f"!{name}/tests/**/*.py",
        ],
        dependencies=kwargs.get("dependencies", ()),
        resolve=resolve,
    )

pex_binary(
        name="pex-binary",
        description=f"PEX application for {name}",
        entry_point=kwargs["binary_entry_point"],
        dependencies=binary_dependencies,
        args=kwargs.get("binary_args", []),
        resolve=resolve,
    )
The errors I am seeing are:
Copy code
17:06:39.39 [WARN] Pants cannot infer owners for the following imports in the target libs/python/app_utils/app_utils/base_kafka_consumer_app.py:../sources:

  * confluent_kafka.Message (line: 12)

If you do not expect an import to be inferable, add `# pants: no-infer-dep` to the import line. Otherwise, see <https://www.pantsbuild.org/2.27/docs/using-pants/troubleshooting-common-issues#import-errors-and-missing-dependencies> for common problems.
17:06:39.49 [WARN] Pants cannot infer owners for the following imports in the target libs/python/kafka_utils/kafka_utils/kafka_schema_serialization.py:../sources:

  * confluent_kafka.schema_registry.Schema (line: 3)
  * confluent_kafka.schema_registry.SchemaRegistryClient (line: 3)
  * confluent_kafka.schema_registry.SchemaRegistryError (line: 3)
  * confluent_kafka.schema_registry.avro.AvroDeserializer (line: 4)
  * confluent_kafka.schema_registry.avro.AvroSerializer (line: 4)
  * confluent_kafka.schema_registry.json_schema.JSONDeserializer (line: 5)
  * confluent_kafka.schema_registry.json_schema.JSONSerializer (line: 5)
  * confluent_kafka.schema_registry.serde.BaseDeserializer (line: 6)
  * confluent_kafka.schema_registry.serde.BaseSerializer (line: 6)
to my understanding having the module mapping should have resolved this but I am still facing this problem despite having defined it.
I am guessing this has something to do with having multiple resolves?
w
w
yeah, that looks like what I am trying to do, mapping the imports to the module with non-standard naming.
h
I think you have the module_mapping backwards. It’s a mapping from “name of distribution” to “list of modules it provides”
So it should be something like 
"confluent-kafka": ["confluent_kafka"],
I think?
And you shouldn’t need to enumerate the submodules, as far as I can remember
w
ill try that! thanks
w
Yeah, as per the link - I assumed what you'd want is more of this form
Copy code
"opentelemetry-api": (
        "opentelemetry._logs",
        "opentelemetry.attributes",
        "opentelemetry.baggage",
        "opentelemetry.context",
        "opentelemetry.environment_variables",
        "opentelemetry.metrics",
        "opentelemetry.propagate",
        "opentelemetry.propagators",
        "opentelemetry.trace",
        "opentelemetry.util",
        "opentelemetry.version",
    ),
f
Would be great to get mappings for confluent-kafka into the default mapper!
h
True, @witty-breakfast-55011 if you find a combo that works could you update the defaults in src/python/pants/backend/python/dependency_inference/default_module_mapping.py ? It would be a quick PR, and we’d appreciate it!
w
sure thing, created one here
2 Views
Open in Slack
PreviousNext
Powered by