Hey all, There’s an issue with pants not able to find pkg_resources when torch is a dependency. Here are the other relevant dependencies in our pyproject.toml (used by poetry): before:

torch=1.10.1
fsspec=0.8.7
pytorch_lightning=1.2.7
torchmetrics=0.3.2
hypothesis=6.30.1
matplotlib=3.5.0

While upgrading pytorch_lightning (to 1.5.0) and torchmetrics (0.6.0) we had dependency conflicts due to fsspec (2021.11.1) . When trying to import torch and running it through pants we got a “No module named pkg_resources found” error. We however were able to import torch on a virtualenv with the same exact modules installed without any error.

_______ ERROR collecting file/path/test.py ________
ImportError while importing test module '/tmp/process-execution9BggO8/file/path/test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/home/shreyas/.pyenv/versions/3.7.12/lib/python3.7/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
file/path/test.py:4: in <module>
    import torch
/home/shreyas/.cache/pants/named_caches/pex_root/venvs/short/8b6cc6fb/lib/python3.7/site-packages/torch/__init__.py:29: in <module>
    from .torch_version import __version__ as __version__
/home/shreyas/.cache/pants/named_caches/pex_root/venvs/short/8b6cc6fb/lib/python3.7/site-packages/torch/torch_version.py:3: in <module>
    from pkg_resources import packaging  # type: ignore[attr-defined]
E   ModuleNotFoundError: No module named 'pkg_resources'

See https://www.pantsbuild.org/docs/python-third-party-dependencies#requirements-with-undeclared-dependencies It's almost certainly the case that pytorch has a missing dep bug.

Thanks, trying this. cc: @elegant-vase-77121

Did this work out?

I’ve to update to poetry pants 2.8 for this right? I’m on 2.5 edit: meant to write pants

To use the overrides field, yes, you need to upgrade to pants 2.8. The workaround is to remove the problematic requirement from your requirements.txt and use an explicit python_requirement target with

dependencies

set manually. (Target was formerly called python_requirement_library)

Got it, thanks

@hundreds-father-404 - if we use

overrides

, does it mean that we can leave torch in

requirements.txt

?

yes, you're changing how the

python_requirement

target gets generated from the

requirements.txt

Got it, thanks for the quick turnaround!

you're welcome! check out https://www.pantsbuild.org/docs/upgrade-tips for upgrade tips. 2.8 adds

update-build-files

to automate some of it also the release blogs if you haven't yet seen them https://blog.pantsbuild.org

🎉