Hi, I have just upgraded from 2.10 to 2.14 pants. We used to have empty BUILD files earlier. Somewhat like

python_sources()

python_tests(
name="tests",
)

And pants used to figure out all the dependencies. But now after the upgrade, all the test cases with above BUILD files are failing since pants is not able to figure out the dependencies. Is there a setting I am missing or this is no longer supported?

Hi, sorry for the trouble. Dependency inference is definitely still a major feature of Pants, so obviously something is awry. Are you seeing any warnings logged?

Example of failing test case. This below was working earlier:

python_tests(
    name="dummy_test.py",
    sources=["test_dummy.py"],
    dependencies=[":dum"],
)

But now it throws exception:

ModuleNotFoundError: No module named 'pkg_resources'

Even though I have this in my project BUILD file

python_requirement(
    name="setuptools",
    requirements=["setuptools"],
)

My pants.toml looks like:

[python]
enable_resolves = true
resolves_generate_lockfiles = false

[python.resolves]
python-default = "requirements.txt"

I ran .

/pants filter --target-type=python_requirement ::

and see //:reqs#setuptools //:setuptools

What are the 3rd party packages being used in your dummy_test?

Yeah, though I have two virtualenvs one with poetry 1.1.14, pants 2.10 and another with poetry 1.3.1 and pants 2.14. I have not added explicit "`:reqs#setuptools`" in both `python_tests`() but have

python_requirement

with setuptools in project level BUILD file. Only other changes are in pants.toml. Now the test in first environment is working, whereas the second environment with upgraded pants is failing with below error.

E   ModuleNotFoundError: No module named 'pkg_resources'

I would run it with same version of poetry to just rule out if it’s something related to poetry version itself.

I think I see the problem

requirements.txt is created manually via

poetry export -f requirements.txt -o requirements.txt  --without-hashes

So you're using a poetry-generated

requirements.txt

as your lockfile. What is the BUILD file target that wraps that? And is

pytorch-lightning

in that requirements.txt?

Yep, that's correct. We are using poetry-generated requirements.txt as a lockfile. Here's my pants.toml config:

[poetry]
version = "poetry==1.3.1"

[python]
interpreter_constraints = ["==3.7.*"]
enable_resolves = true
resolves_generate_lockfiles = false

[python.resolves]
python-default = "requirements.txt"

[pytest]
extra_requirements = ["pytest-mock==3.8.2", "pytest-pgsql==1.1.2", "pytest-icdiff==0.6", "pygments==2.12.0", "pytest-cov==3.0.0", "pytest-bdd==6.0.1"]
lockfile = "pytest_lockfile.txt"

And this is my project level BUILD file.

python_requirement(
    name="torch",
    requirements=["torch==1.10.0+cpu"],
    dependencies=["//:reqs#setuptools"],
)

python_requirement(
    name="pytorch-lightning",
    requirements=["pytorch-lightning"],
    dependencies=[":torch"],
)

python_requirements(
    name="reqs",
    module_mapping={
        "python-constraint": ["constraint"],
        "google-api-core": ["google.api_core"],
        "grpcio": ["grpc"],
        "grpcio-health-checking": ["grpc_health"],
        "python-json-logger": ["pythonjsonlogger"],
    },
)

requirements.txt

doesn't contain torch or pytorch-lightning, these are only present in BUILD file.

Gotcha. So that explains the problem. The lockfile must include all the requirements you're using... That's easy to achieve when Pants generates the lockfiles for you, but if using external lockfiles, as you are, you have to keep them in sync manually.

Thanks @happy-kitchen-89482. Does that mean adding

pytorch-lightning==1.5.0

as a python_requirement in BUILD file is same as adding adding

pytorch-lightning==1.5.0 ; python_version >= "3.7" and python_version < "4.0"

in requirements.txt. Another question I have is, does pants try to download the requirement mentioned in BUILD file's python_requirement via pip? In above case, should torch and pytorch-lightning be added to requirements.txt to resolve the issue? And if I do add them then is there a use of having them in BUILD file at all?

Not quite. The BUILD files are necessary, they tell Pants what requirements you want to make available to your code.

Thanks Benjy, a quick clarification. You mean pants to generate lock files via:

./pants generate-lockfiles

, right?

Exactly