https://pantsbuild.org/ logo
#general
Title
# general
l

lively-exabyte-12840

02/28/2022, 4:20 PM
Sanity check - I've got two modules - snowflake-connector-python and snowflake-sqlalchemy - that both map to the name "snowflake". I wasn't able to use python_requirements/module_mapping to express that; I found that only one of those could be mapped to "snowflake", with the last one winning. But I was able to do what's pasted below - that seem like the best approach?
Copy code
python_requirement(
    name="snowflake",
    requirements=[
        "snowflake-connector-python==2.7.2",
        "snowflake-sqlalchemy==1.3.2"
    ],
    modules=["snowflake"]
)
b

bitter-ability-32190

02/28/2022, 4:22 PM
Oh that's interesting. So both packages get unpacked into a
snowflake
directory in the venv? And they just agree not to step on each others toes? Last I checked
poetry
does this as well with
poetry-core
going into
poetry/core
.
l

lively-exabyte-12840

02/28/2022, 4:26 PM
Right - the toplevel.txt file in both wheels has a value of "snowflake"
🤔 1
b

bitter-ability-32190

02/28/2022, 4:27 PM
Without thinking of the technical feasibility, seems like something that should be supported by Pants. In the meantime just want to double-check your posted workaround works for you?
Either way, would you mind filing an issue? https://github.com/pantsbuild/pants/issues
l

lively-exabyte-12840

02/28/2022, 4:29 PM
It does work. I'll file an issue too.
1
Just curious, I was surprised by the module_mapping syntax - that it's "package name": [array of mapped names]. I haven't thought of a use case when I'd want to map a package name to multiple names... but here I have a use case where I'd want to map multiple package names to the same logical name
b

bitter-ability-32190

02/28/2022, 4:39 PM
FWIW there are some (admittedly rare) cases that actually map a package to multiple modules. I think logically speaking it makes sense to map packages to modules (as that's semantically what packages are/do). But you bring up a very interesting point that the reverse might make more sense in this context. "These packages provide these modules" vs "These modules are provided by these packages" @hundreds-father-404 might have thoughts as well
e

enough-analyst-54434

02/28/2022, 4:42 PM
FWIW this is canonical for projects with namespace packages. The very purpose being to support distribution in multiple artifacts.
🙌 1
1
Like Pants itself!
b

bitter-ability-32190

02/28/2022, 4:43 PM
Ok so admittedly much less rare 😛
e

enough-analyst-54434

02/28/2022, 4:44 PM
Especially since historically the big3 cloud manager tools all used this extensively. Each had 10s to near 100s of dists contributing to aws, azure, google_cloud? namespace packages.
In all these cases though, generally one layer down is not shared, SO being able to module map dist1 -> "aws.s3", dist2 -> "aws.ec2", etc - would be beneficial for pulling in just the deps you need insteal of all 100!
b

bitter-ability-32190

02/28/2022, 4:47 PM
That does make me realize that the module mapper allows for qualified module names: https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/python/dependency_inference/default_module_mapping.py
@lively-exabyte-12840 do either of the packages you listed install to a submodule of the top-level one? That might work
e

enough-analyst-54434

02/28/2022, 4:48 PM
b

bitter-ability-32190

02/28/2022, 5:00 PM
I have that link in my clipboard, but for whatever reason didn't paste it 😐
h

hundreds-father-404

02/28/2022, 5:43 PM
Yeah that indeed is what is intended. For Pants's own distributions, we set
pantsbuild.pants
to be
pants
module whereas
pantsbuild.pants.testuil
is
pants.testutil
- our dep inference code tries the most precise thing first, then falls back to the general one
h

happy-kitchen-89482

02/28/2022, 5:56 PM
And as for distribution packages that map to multiple module names, there are some! E.g.,
setuptools
provides
setuptools
,
easy_install
and
pkg_resources
, and
pymongo
provides
bson
,
gridfs
and
pymongo
2
l

lively-exabyte-12840

02/28/2022, 6:14 PM
@bitter-ability-32190 Both of those packages contain namespace_packages.txt and top_level.txt files that only contain "snowflake".
1