Hello. I have a question on missing dependencies. The simplified story is, I have the following in a python file

a.py

import x
import y
import z
......
......

when I do .

/pants dependencies a.py

. I only see

x

and

y

, but not

z

. Then I started to investigate more. First, If I do

./pants list :

, I do see

//:z

, which suggests me that pants does know

z

which is from

requirements.txt

. Second, I add

--no-process-execution-local-cleanup

flag to the

./pants dependencies

, then go to the cached folder. When I run

__run.sh

in that folder, I do see

z

in the output. It suggest me that

z

should be treated as a dependency in

a.py

. I am wondering why

./pants dependencies a.py

does not show

z

. I took a look at https://www.pantsbuild.org/docs/troubleshooting#import-errors-and-missing-dependencies but i am not sure if any of them applies here.

Is the name of

z

the same in requirements.txt as in the import?

grpcio

, for example, you have to import as

grpc

- so then a

module_mapping

is needed for Pants to properly wire it into the dependency graph.

yes. it is the same name.

Does

./pants dependencies --transitive

show that z is included, perhaps?

no for the last two questions. I am trying to reproduce this error on a simple repo.

This question depends on a) which Pants version you're using? and b) what the path is to requirements.txt is? You might be hitting a really confusing issue fixed in Pants 2.10

I am still facing this issue. I am using 2.8. The requirements.txt lives in the root. If I do

./pants dependencies --type=3rdparty ::

. I can see other dependencies rather than

z

. Note that

z

is a new dependency just got added.

what is the answer to the above two questions? You might need to add the path to

[GLOBAL].pantsd_invalidation_globs

Thanks @hundreds-father-404 ! One missing module I have now is netCDF4, which does not seem require model_mapping. I could be wrong. Pants does not catch it for some reasons. All my other dependencies seems fine.

is it showing up with

./pants list ::

?

yes. but it just does not show in .pants dependencies

so if you manually add the dependency, do things work? I'm trying to make sure this is specifically an issue with dependency inference

I will try later today and let you know

Hi @hundreds-father-404, I am working with @wide-zoo-86070 and we are able to “fix” the issue by manually adding dependency. foo.py:

import netCDF4

BUILD

python_tests(
    name="tests",
    dependencies=[
        "//:root#netCDF4",
    ],
)

Let us know if helps to understand the bug better. Other dependencies were auto-detected as expected.

Great! So then that confirms it is an issue with dependency inference in particular. 1. What does the import of netCD4 look like?

import netcd4

for example? 2. Do you define

netCD4

multiple times in your repository?

1. import netCDF4 2. netCDF is defined in requirements.txt as: netCDF4==1.5.8

I think you said you already set

module_mapping

, but to double check, could you try adding

module_mapping={'netCD4': ["netcd4"]}

to the

python_requirements

target? Then remove the explicit dep for now and run

./pants dependencies

to see if it worked

No, as soon as I remove explicit dependency it stops working BUILD (in the git root)

python_requirements(
    name="root",
    module_mapping={'netCDF4': ["netcdf4"]}
)

Some issue with case-sensitive module name?

I went through a lot of trouble to try to make sure this is case insensitive for

module_mapping

, but it's worth experimenting with! Note that you can list as many modules as you want also in that list And to be crystal clear, no warning from Pants about ambiguous dependencies, right?

(checked https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/python/dependency_inference/default_module_mapping.py) There is similar issue with “pypdf2”: (“PyPDF2",),

ughhhh this is supposed to handle capitalization 🤦 would you be willing to please open a ticket at https://github.com/pantsbuild/pants/issues/new/choose?

And to be crystal clear, no warning from Pants about ambiguous dependencies, right?

Have not seen any

oh wait!

it is:

import netCDF4

K, Pants is behaving how we want. The capitalization does matter for the module you import; what should not matter is the key of the dictionary:

module_mapping={'netcd4': ['netCD4']},

should work and so should

module_mapping={'netCD4': ['netCD4']},

But you need the list to have the right capitilzation. Taht's intentional

hmm… What is a meaning of:

module_mapping={'netCD4': ['netCD4']},

map A to A

Pants's default is to lowercase everything, so that if your requirements.txt has

Django

, we assume the module is

django

. Most of the time, that's the best default. Here, it is not