https://pantsbuild.org/ logo
Join Slack
Powered by
hey guys, trying to figure out how to get `mypy` t...
# general
h
hey guys, trying to figure out how to get 
mypy
to use 
types-beautifulsoup4
stubs but I can’t seem to get it configured properly 🧵
using poetry, I can run mypy successfully using 
poetry run mypy src
with the following dev dependencies declared in `pyproject.toml`:
Copy code
[tool.poetry.group.dev.dependencies]
mypy = "^1.4.0"
types-beautifulsoup4 = "^4.12.0.5"
however, I can’t get mypy configured correctly in pants (2.16), it keeps complaining about missing stubs:
Copy code
$ pants check ::
...
10:55:20.00 [ERROR] Completed: Typecheck using MyPy - mypy - mypy failed (exit code 2).
src/example.py:9: error: Library stubs not installed for "bs4"  [import]
src/example.py:10: error: Library stubs not installed for "bs4.element"  [import]
src/example.py:10: note: Hint: "python3 -m pip install types-beautifulsoup4"
src/example.py:10: note: (or run "mypy --install-types" to install all missing stub packages)
src/example.py:10: note: See <https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports>
Found 2 errors in 1 file (checked 4 source files)
here’s the relevant part of my 
pants.toml
Copy code
[python.resolves]
mypy = "3rdparty/python/mypy.lock"

[mypy]
install_from_resolve = "mypy"
requirements =["//3rdparty/python:mypy"]
and 
3rdparty/python/mypy-requirements.txt
Copy code
mypy>=1.0,<2.0
types-beautifulsoup4>=4.12.0.5,<5.0
types-pytz>=2023.3.0.0,<2024.0
types-requests>=2.31.0.1,<3.0
and 
3rdparty/python/BUILD
Copy code
python_requirements(
    name="mypy",
    source="mypy-requirements.txt",
    resolve="mypy",
)
I don’t have a 
mypy.ini
set up, but using 
config = "mypy.ini"
and the following 
mypy.ini
Copy code
[mypy]
plugins =
  bs4-stubs
gives me a different error
Copy code
$ pants check ::
...
11:09:10.75 [ERROR] Completed: Typecheck using MyPy - mypy - mypy failed (exit code 2).
Found 1 error in 1 file (errors prevented further checking)

mypy.ini:2: error: Plugin "bs4-stubs" does not define entry point function "plugin"  [misc]
what am i doing wrong here that mypy can’t find the stubs I’m giving it in the requirements file?
b
I'm pretty sure Pex is just grabbing mypy from your resolve, not the rest
You might have to make the mypy requirement depend on the types requirements inside the`python_requirements` target
h
ah ok, so it’s not using the entire 
mypy.lock
/ 
mypy-requirements.txt
for 
mypy
then? only 
mypy
specifically, and it’s ignoring the remaining reqs i have in the requirements file unless I specifically tell it to include them?
whatever the issue is, it seems to be related to the stubs for beautifulsoup4 specifically, i don’t think the others are having any errors
unfortunately, i’ve tried both 
overrides
and 
type_stubs_module_mapping
, and neither seem to work:
Copy code
python_requirements(
    name="mypy",
    source="mypy-requirements.txt",
    resolve="mypy",
    type_stubs_module_mapping={"types-beautifulsoup4": ["bs4"]},
    overrides={
        "mypy": {
            "dependencies": ["3rdparty/python:mypy#types-beautifulsoup4"],
        },
    }
)
b
The easiest way to poke around is 
pants --keep-sandboxes=on_failure ...
and then poke at the sandbox
🙏 1
h
thanks! apologies for the dumb question, but what exactly am I looking for? I can see a 
mypy.pex/PEX-INFO
file in the sandbox which would seem to me to indicate that 
types-beautifulsoup4
is included:
Copy code
{
  "bootstrap_hash": "774e132021b0a8c19e8a935452a4415d5ec16f93",
  "build_properties": {
    "pex_version": "2.1.134"
  },
  "code_hash": "f3331d84c5b30eaf04417cd3fc0dc95864d7937f",
  "distributions": {
    "mypy-1.4.1-cp311-cp311-macosx_11_0_arm64.whl": "b06c414ddea0c6df5d6cfa741320a8dc78150754ece26d7be790a48ce586c208",
    "mypy_extensions-1.0.0-py3-none-any.whl": "322faffb2e4c7768d03d354528240ded8adea9dd0e6e67d998f275994c744cb5",
    "types_beautifulsoup4-4.12.0.5-py3-none-any.whl": "64549893d677f5e3d2852ae45dbdda1a0d6ff63a1163db239acf9f4b2362ce7b",
    "types_html5lib-1.1.11.14-py3-none-any.whl": "8e8e95bd3b28462ed28c210b87535163b8a0ececc162384cf17cd48e6e77b93a",
    "types_pytz-2023.3.0.0-py3-none-any.whl": "77afe64369f5fd5368aa7ab394a45bee5fdd4b2a76c778994fa5d0cd5c0f0c94",
    "types_requests-2.31.0.1-py3-none-any.whl": "f5f50d985bd991f221b7cb5ee2777e45325b574d974042ace3d0744ab6a54273",
    "types_urllib3-1.26.25.13-py3-none-any.whl": "7aca16f70169e31fbd943b512c72b50d06ba6f33e31144c33c127f53718154dc",
    "typing_extensions-4.7.1-py3-none-any.whl": "5e6df9210d60789205e617f60bb181074200092ebed2796a63358a9c9f3f1c75"
  },
  "emit_warnings": false,
  "entry_point": "mypy.__main__:console_entry",
  "ignore_errors": false,
  "includes_tools": true,
  "inherit_path": "false",
  "inject_args": [],
  "inject_env": {},
  "interpreter_constraints": [],
  "pex_hash": "ff22d350f24b4d9d4161cc7a3d91455d58af1f08",
  "pex_path": "",
  "pex_paths": [],
  "requirements": [
    "mypy<2.0,>=1.0",
    "types-beautifulsoup4<5.0,>=4.12.0.5",
    "types-pytz<2024.0,>=2023.3.0.0",
    "types-requests<3.0,>=2.31.0.1"
  ],
  "strip_pex_env": true,
  "venv": true,
  "venv_bin_path": "false",
  "venv_copies": false,
  "venv_hermetic_scripts": true,
  "venv_site_packages_copies": true
}
b
I'm wondering if 
mypy
isn't checking its own env for requirements? And just checking the env of the provided Python
h
hmm. if i understood you correctly, i think you might be onto something. I added an override to my root BUILD file and mypy was able to resolve correctly:
Copy code
poetry_requirements(
    name="root",
    overrides={
        "beautifulsoup4": {
            "dependencies": ["//:root#types-beautifulsoup4"],
        },
    }
)
b
Yeah I suppose that makes sense. Mypy trying to isolate itself from the environment its checking
h
this doesn’t seem to work if i add it to just the mypy target:
Copy code
python_requirements(
    name="mypy",
    source="mypy-requirements.txt",
    resolve="mypy",
    overrides={
        "beautifulsoup4": {
            "dependencies": ["3rdparty/python:mypy#types-beautifulsoup4"],
        },
    }
)
gives
Copy code
$ pants check ::
17:08:07.38 [ERROR] 1 Exception encountered:

Engine traceback:
  in `check` goal
  in Find targets from input specs

InvalidFieldException: Unused key in the `overrides` field for 3rdparty/python:mypy: ['beautifulsoup4']
b
That makes sense, 
beautifulsoup
isn't a top-level requirement in your reqs file. I don't think that'd work as expected either. Anyways, seems like you have enough to go from here?
h
this feels dirty though lol, i shouldn’t need to include the stubs whenever i’m importing bs4
That makes sense, 
beautifulsoup
isn’t a top-level requirement in your reqs file.
you mean adding 
beautifulsoup4
to 
mypy-requirements.txt
? unfortunately that seems to give me the same error as before for the missing library stubs this seems functional, if not the most elegant solution, but I’ll take what I can get. so your recommendation would be to just add it to the overrides in the root dependencies then?
b
I'm not terribly sure. I mean it works, but I agree it seems non-ideal 😕
I'm wondering if the 
extra_type_stubs
deprecation should be un-deprecated? https://www.pantsbuild.org/docs/reference-mypy#extra_type_stubs) There's also this: https://www.pantsbuild.org/docs/reference-python_requirement#codetype_stub_modulescode so maybe try that?
h
hmm yeah, the 
extra_type_stubs
seemed to do the trick! so now I have 
pants.toml
:
Copy code
[mypy]
install_from_resolve = "mypy"
extra_type_stubs = [
  "types-beautifulsoup4",
]
requirements =["//3rdparty/python:mypy"]
3rdparty/python/mypy-requirements.txt
:
Copy code
mypy>=1.0,<2.0
types-beautifulsoup4>=4.12.0.5,<5.0
types-pytz>=2023.3.0.0,<2024.0
types-requests>=2.31.0.1,<3.0
`3rdparty/python/BUILD`:
Copy code
python_requirements(
    name="mypy",
    source="mypy-requirements.txt",
    resolve="mypy",
)
seems cleaner at least 🙂
b
FWIW that option is deprecated and will be removed
Can you try the second option? That appears to be non-deprecated 😅
😂 1
I think you'll need to move those requirements into your user "resolve" (like the one for your code)
h
type_stub_modules
seems to be exclusive to 
python_requirement
and not 
python_requirements
fwiw if i’m doing it right, it doesn’t seem to work:
Copy code
python_requirement(
    name="mypy",
    requirements=[
        "mypy>=1.0,<2.0",
        "types-beautifulsoup4>=4.12.0.5,<5.0",
        "types-pytz>=2023.3.0.0,<2024.0",
        "types-requests>=2.31.0.1,<3.0",
    ],
    resolve="mypy",
    type_stub_modules=["types-beautifulsoup4"]
)
b
python_requirements
is a generator generating 
python_requirement
targets and you'd need all this on the right target(s) for your normal user resolve. Not the mypy one
And that resolve would need to contain these packages
h
if i’m following, wouldn’t that mean that all 4 packages are always included in the 
mypy
target? it still complains about the missing stub when using it this way
b
This thread is very timely for me. I've tried a few different combinations to get stubs loaded as discussed here and pouring over the documents and the only thing I could get to work is the deprecated 
extra_type_stubs
in the 
pants.toml
file. Hopefully this gets flushed out a bit before deprecation.
b
Sorry, didn't respond to the last message. The requirements would be on your user resolve NOT the mypy resolve. E.g. where your bs4 requirement is
i
I just hit this as well upgrading from 2.15 to 2.16. I had to keep the 
extra_type_stubs
(and their lockfile) in place for mypy to continue working - putting the stubs in the mypy resolver resulted both in them being included in the mypy pex (verified in the sandbox), and not being used during checking
h
yeah, that seems to work but the issue is that option is deprecated and removed in 2.18 😕
d
he requirements would be on your user resolve NOT the mypy resolve. E.g. where your bs4 requirement is
This works for me
b
Hey all! I'm trying to upgrade from 2.15 to 2.20 and running into this same issue with mypy not having 
extra_type_stubs
anymore. I've set up a mypy resolve based on the lockfile docs, and I can see that it's correctly resolving the dependencies when running 
check
, but I still get 
error: Library stubs not installed for <various libs>  [import]
d
@best-furniture-15331 , you have separate requirements.txt for MyPy? Can you share what's your exact setup, i.e. pants.toml section for MyPy?
b
@dry-architect-80370 thanks for the quick response! let me know if you need more than this: 
pants.toml
Copy code
[GLOBAL]
pants_version = "2.20.0"
backend_packages = [
  "pants.backend.python",
  "pants.backend.python.lint.autoflake",
  "pants.backend.python.lint.black",
  "pants.backend.python.lint.isort",
  "pants.backend.python.lint.flake8",
  "pants.backend.python.typecheck.mypy",
  "pants.backend.google_cloud_function.python",
  "pants.backend.shell",
  "pants.backend.shell.lint.shellcheck",
  "pants.backend.shell.lint.shfmt",
  "pants.backend.codegen.protobuf.python",
  "pants.backend.docker",
]

[python]
enable_resolves = true
default_resolve = "common"
interpreter_constraints = ["CPython==3.9.*"]

[python.resolves]
common = "3rdparty/python/common.lock"
mypy = "3rdparty/python/mypy.lock"

[mypy]
install_from_resolve = "mypy"
requirements = ["//3rdparty/python:mypy"]

[python-protobuf]
mypy_plugin = true
3rdparty/python/BUILD
Copy code
python_requirements()

python_requirements(
    name="mypy",
    source="mypy-requirements.txt",
    resolve="mypy",
)
3rdparty/python/mypy-requirements.txt
Copy code
mypy==1.5.0
types-protobuf==3.20.2
types-redis==4.3.13
types-requests==2.28.0
d
Remove the 
requirements
entry from the 
mypy
section of 
pants.toml
. In your current setup you are telling Pants that you only need the 'mypy' requirement from that resolve, and it discards the rest.
b
That's how I had it previously, but I changed it based on the docs Regardless, I tried removing the 
requirements
entry again and re-ran but same error:
Copy code
>> pants --changed-since=origin/main --changed-dependents=transitive check
[...]
error: Library stubs not installed for "redis"
[...]
I also confirmed that e.g. 
types-redis
is in the 
mypy.lock
file
Copy code
[...]
"requirements": [
    "mypy==1.5.0",
    "types-protobuf==3.20.2",
    "types-redis==4.3.13",
    "types-requests==2.28.0",
  ],
[...]
d
Seems like things are not working like described in the docs:
Copy code
If you install MyPy from a custom lockfile you can also add type stub requirements to that lockfile. This ensures that the stubs are only used when running MyPy and are not included when, for example, packaging a PEX.
I have checked, and the same thing happens to me, I need to add 
types-redis
to the default requirements (not only the MyPy ones) to make it work. Minimal repro: https://github.com/kordek/pants-mypy-reqs-repro
👀 2
b
so currently in 2.20, the only way to get type stubs into mypy is through the 
common
requirements file?
👀 1
s
Any confirmation on this? I'm following along, but after adding stubs directly in my requierments.txt I'm still getting mypy errors from failing to install the stubs: requirements.txt:
Copy code
...
transformers==4.40.2
types-protobuf==3.20.3
types-psycopg2==2.9.5
types-PyYAML==6.0.12.2
types-redis==4.3.13
types-requests==2.28.0
types-setuptools==57.4.4
uvicorn==0.21.0
When running:
Copy code
pants --changed-since=origin/main --changed-dependents=transitive check

...
src/python/common/db/records/paper_records_test.py:9:1: error: Library stubs not installed for "google.protobuf.json_format"  [import]
    from google.protobuf.json_format import MessageToJson
Running 
pants dependencies --transitive src/python/common/db/records/paper_records_test.py
, I see that types-protobuf is not inferred:
Copy code
3rdparty/python#SQLAlchemy
3rdparty/python#aiopg
3rdparty/python#loguru
3rdparty/python#protobuf
3rdparty/python#psycopg2-binary
3rdparty/python#pyarrow
3rdparty/python#pydantic
3rdparty/python#testcontainers
3rdparty/python#types-psycopg2
3rdparty/python/requirements.txt
src/infra/db:main_sql
src/protos/paper_metadata.proto
src/python/common/beam/records/base_record.py
src/python/common/config/__init__.py
src/python/common/config/constants.py
src/python/common/db/__init__.py
src/python/common/db/execute.py
src/python/common/db/papers_util.py
src/python/common/db/records/paper_records.py
src/python/common/db/test/connect.py
src/python/common/time/convert.py
m
also encountered a similar issue. there's a related github issue: https://github.com/pantsbuild/pants/issues/21114 do we have plans to address it any time soon? it's preventing us from using 
pants check
24 Views
Open in Slack
PreviousNext
Powered by