https://pantsbuild.org/ logo
#general
Title
# general
w

wide-zoo-86070

03/17/2022, 3:07 PM
Hello. I have a question on missing dependencies. The simplified story is, I have the following in a python file
a.py
Copy code
import x
import y
import z
......
......
when I do .
/pants dependencies a.py
. I only see
x
and
y
, but not
z
. Then I started to investigate more. First, If I do
./pants list :
, I do see
//:z
, which suggests me that pants does know
z
which is from
requirements.txt
. Second, I add
--no-process-execution-local-cleanup
flag to the
./pants dependencies
, then go to the cached folder. When I run
__run.sh
in that folder, I do see
z
in the output. It suggest me that
z
should be treated as a dependency in
a.py
. I am wondering why
./pants dependencies a.py
does not show
z
. I took a look at https://www.pantsbuild.org/docs/troubleshooting#import-errors-and-missing-dependencies but i am not sure if any of them applies here.
n

narrow-vegetable-37489

03/17/2022, 3:13 PM
Is the name of
z
the same in requirements.txt as in the import?
grpcio
, for example, you have to import as
grpc
- so then a
module_mapping
is needed for Pants to properly wire it into the dependency graph.
w

wide-zoo-86070

03/17/2022, 3:13 PM
yes. it is the same name.
h

hundreds-father-404

03/17/2022, 5:02 PM
Does
./pants dependencies --transitive
show that z is included, perhaps?
no warning about ambiguous dependency inference because the module defined by
z
is defined by >1
python_requirement
target?
w

wide-zoo-86070

03/17/2022, 8:05 PM
no for the last two questions. I am trying to reproduce this error on a simple repo.
👍 1
@hundreds-father-404 i have a similar problem. support I recently add one dependency in requirements.txt, then use this dependency in a python module. After that, if i run
./pants dependencies that-module.py
, will the new dependencies automatically shows up? or I have to something before it?
h

hundreds-father-404

03/17/2022, 9:08 PM
This question depends on a) which Pants version you're using? and b) what the path is to requirements.txt is? You might be hitting a really confusing issue fixed in Pants 2.10
😅 1
w

wide-zoo-86070

03/22/2022, 5:22 PM
I am still facing this issue. I am using 2.8. The requirements.txt lives in the root. If I do
./pants dependencies --type=3rdparty ::
. I can see other dependencies rather than
z
. Note that
z
is a new dependency just got added.
h

hundreds-father-404

03/24/2022, 1:33 AM
what is the answer to the above two questions? You might need to add the path to
[GLOBAL].pantsd_invalidation_globs
w

wide-zoo-86070

03/24/2022, 3:55 AM
Thanks @hundreds-father-404 ! One missing module I have now is netCDF4, which does not seem require model_mapping. I could be wrong. Pants does not catch it for some reasons. All my other dependencies seems fine.
h

hundreds-father-404

03/24/2022, 5:02 AM
is it showing up with
./pants list ::
?
w

wide-zoo-86070

03/24/2022, 2:31 PM
yes. but it just does not show in .pants dependencies
h

hundreds-father-404

03/24/2022, 4:45 PM
so if you manually add the dependency, do things work? I'm trying to make sure this is specifically an issue with dependency inference
w

wide-zoo-86070

03/24/2022, 5:39 PM
I will try later today and let you know
👍 1
c

curved-electrician-64177

03/25/2022, 6:47 PM
Hi @hundreds-father-404, I am working with @wide-zoo-86070 and we are able to “fix” the issue by manually adding dependency. foo.py:
Copy code
import netCDF4
BUILD
Copy code
python_tests(
    name="tests",
    dependencies=[
        "//:root#netCDF4",
    ],
)
Let us know if helps to understand the bug better. Other dependencies were auto-detected as expected.
h

hundreds-father-404

03/25/2022, 6:48 PM
Great! So then that confirms it is an issue with dependency inference in particular. 1. What does the import of netCD4 look like?
import netcd4
for example? 2. Do you define
netCD4
multiple times in your repository?
c

curved-electrician-64177

03/25/2022, 6:49 PM
1. import netCDF4 2. netCDF is defined in requirements.txt as: netCDF4==1.5.8
h

hundreds-father-404

03/25/2022, 6:50 PM
I think you said you already set
module_mapping
, but to double check, could you try adding
module_mapping={'netCD4': ["netcd4"]}
to the
python_requirements
target? Then remove the explicit dep for now and run
./pants dependencies
to see if it worked
c

curved-electrician-64177

03/25/2022, 6:56 PM
No, as soon as I remove explicit dependency it stops working BUILD (in the git root)
Copy code
python_requirements(
    name="root",
    module_mapping={'netCDF4': ["netcdf4"]}
)
Some issue with case-sensitive module name?
But… this mapping works:
Copy code
python_requirements(
    name="root",
    module_mapping={'netcdf4': ["netCDF4"]}
)
😅 1
1
h

hundreds-father-404

03/25/2022, 6:57 PM
I went through a lot of trouble to try to make sure this is case insensitive for
module_mapping
, but it's worth experimenting with! Note that you can list as many modules as you want also in that list And to be crystal clear, no warning from Pants about ambiguous dependencies, right?
c

curved-electrician-64177

03/25/2022, 6:58 PM
h

hundreds-father-404

03/25/2022, 6:58 PM
ughhhh this is supposed to handle capitalization 🤦 would you be willing to please open a ticket at https://github.com/pantsbuild/pants/issues/new/choose?
c

curved-electrician-64177

03/25/2022, 6:58 PM
And to be crystal clear, no warning from Pants about ambiguous dependencies, right?
Have not seen any
h

hundreds-father-404

03/25/2022, 6:59 PM
oh wait!
so to be clear, the Python import is
import netCD4
, not
import netcd4
?
c

curved-electrician-64177

03/25/2022, 7:00 PM
it is:
Copy code
import netCDF4
h

hundreds-father-404

03/25/2022, 7:01 PM
K, Pants is behaving how we want. The capitalization does matter for the module you import; what should not matter is the key of the dictionary:
Copy code
module_mapping={'netcd4': ['netCD4']},
should work and so should
Copy code
module_mapping={'netCD4': ['netCD4']},
But you need the list to have the right capitilzation. Taht's intentional
Maybe the docs for
module_mapping
could be more clear with this? Wdyt?
c

curved-electrician-64177

03/25/2022, 7:04 PM
hmm… What is a meaning of:
Copy code
module_mapping={'netCD4': ['netCD4']},
map A to A
h

hundreds-father-404

03/25/2022, 7:27 PM
Pants's default is to lowercase everything, so that if your requirements.txt has
Django
, we assume the module is
django
. Most of the time, that's the best default. Here, it is not
👍 1