https://pantsbuild.org/ logo
Join Slack
Powered by
Got an interesting dependency resolution issue. I ...
# general
h
Got an interesting dependency resolution issue. I added boto3-stubs to get better type hinting in my editor and now pants is thinking 
boto3
maps to 
boto3-stubs
and gets confused on imports. I can go manually add this dependency since it doesn't show up in too many places, but I'm surprised it's getting confused. I even updated my 
module_mapping
to try to force the resolution, but it wasn't happy.
I can open a bug ticket if there's not something quick that comes to mind.
In my module, I have something like
Copy code
import boto3
import click
and dependencies is showing
Copy code
//:boto3-stubs
//:click
I have
Copy code
boto3==1.17.0
botocore>=1.20.0,<2
in a 
requirements.txt
and
Copy code
boto3-stubs>=1.21.0,<2
boto3-stubs[s3,ecr,sts]
in a 
requirements-dev.txt
I found this here
If the dependency is a type stub, and the default does not work, set 
type_stub_modules
 on the 
python_requirement
 target, and 
type_stubs_module_mapping
 on the 
python_requirements
 and 
poetry_requirements
 macros. (The default for type stubs is to strip off 
types-
, 
-types
, 
-stubs
, and 
stubs-
. So, 
types-requests
 gives type stubs for the module 
requests
.)
But that just seems helpful for registering a stub library with another module and not undoing the issue I'm seeing.
h
@hundreds-father-404 is the expert here, but he's out until Friday. Let me see if I can figure it out.
h
Thanks. Another thing I was thinking is that it would be nice if type stubs were only added for goals that actually needed them (e.g. lint). Seems inefficient to package type stubs along with a pex binary (which is the impression I'm getting from running 
dependencies
) since they'll never affect functionality.
Sounds worth a bug ticket so I'll get on that. Edit: ticket here
🙏 1
h
Thanks!
h
@high-yak-85899 why define 
boto3-stubs
twice?
Copy code
boto3-stubs>=1.21.0,<2
boto3-stubs[s3,ecr,sts]
Could you do 
boto3-stubs[s3,ecr,sts]>=1.21.0,<2
instead? I only reproduce the issue when you have both stubs defined, and it's because it triggers ambiguity with dependency inference so Pants can't figure out which you want
Otherwise, things work for me
Copy code
diff --git a/src/python/pants/util/BUILD b/src/python/pants/util/BUILD
index 76d0a50a8..64b634564 100644
--- a/src/python/pants/util/BUILD
+++ b/src/python/pants/util/BUILD
@@ -4,3 +4,6 @@
 python_sources()
 
 python_tests(name="tests")
+
+python_requirement(name="boto3", requirements=["boto3"])
+python_requirement(name="boto3-stubs", requirements=["boto3-stubs"])
diff --git a/src/python/pants/util/collections.py b/src/python/pants/util/collections.py
index 0dd3e3d90..4874ba5d5 100644
--- a/src/python/pants/util/collections.py
+++ b/src/python/pants/util/collections.py
@@ -8,7 +8,10 @@ import collections.abc
 import gc
 import math
 from sys import getsizeof
-from typing import Any, Callable, Iterable, Iterator, MutableMapping, TypeVar
+from typing import Any, Callable, Iterable, Iterator, MutableMapping, TypeVar, TYPE_CHECKING
+
+if TYPE_CHECKING:
+    import boto3
 
 from pants.engine.internals import native_engine
Copy code
❯ ./pants dependencies src/python/pants/util/collections.py
src/python/pants/util:boto3
src/python/pants/util:boto3-stubs
also sorry for the superrrrrr delayed reply to this 😞
h
I didn't try the latter. When you just have 
boto3-stubs
, it only pulls in the most common stubs (which I think didn't include the specific ones I then had to call out).
h
I think 
boto3-stubs[s3,ecr,sts]>=1.21.0,<2
should do the trick
🙌 1
do you know if those two other users huonw and achimnol are on your team? Wondering why they liked the GH issue / if they have the same problem
h
Not that I know of
👍 1
5 Views
Open in Slack
PreviousNext
Powered by