Since moving all of our dependencies into a single resolve, actions like

pants lint

now create, for instance, a

flake8.pex

but attempts to download unneeded packages such as private packages. Our dependencies are spec’d in a

pyproject.toml

with tools et al. being set in separate dependency groups e.g.

[project.optional-dependencies].tools

. How can I make sure that only packages for said tools are downloaded when running

pants lint/fmt

but keep these dependencies spec’d in the same

pyproject.toml

? Our root BUILD file sets the below, which I assume is the reason this behaviour occurs.

python_requirements(
    name="reqs",
    source="pyproject.toml",
    resolve="python-default"
)

I think you want pantsbuild.org/stable/reference/subsystems/flake8#… (requirements and install_from_resolve)

install_from_resolve

tells pants to use flake8 from your lockfile rather than using its own internally pinned version.

requirements

(which probably only needs to be the target of your flake8 requirement) tells pants that these are the only dependencies it needs to run flake8, and then it will know it doesn't need the rest of your lockfile.

@hundreds-carpet-28072 what is your full linter set? You give the example of flake8 implying you don't use ruff as a 1-stop shop. Given that, if you're using

pants.backend.python.lint.pylint

, for example: + Original addition of transitive requirement fetch >6 years ago: github.com/pantsbuild/pants/pull/9794 + Still so: github.com/pantsbuild/pants/blob/…/rules.py#… Assuming Eric wasn't lying in the original PR, direct requirements are needed to have pylint import checking work; so a 3rdparty requirement fetch for source code being linted is needed; although it looks like there has always been the bug that this requirement fetch is transitive instead of intransitive (again - assuming Eric's commentary in the original PR is accurate). > but attempts to download unneeded packages such as private packages. So, assuming I'm on the right track here and you've enabled a plugin like

pylint

that does in fact need at least direct 3rdparty dependencies, the always present bug of fetching transitive 3rdaprty dependencies could maybe explain your observation IFF the private packages are transitive deps of the code being linted and not direct deps.

install_from_resolve

tells pants to use flake8 from your lockfile rather than using its own internally pinned version.

We have

flake8

et al. using the single

python-default

resolve / lockfiles to be clear.

requirements

(which probably only needs to be the target of your flake8 requirement) tells pants that these are the only dependencies it needs to run flake8, and then it will know it doesn’t need the rest of your lockfile.

I would’ve assumed Pants would discern that

flake8.pex

wouldn’t need most other dependencies itself as it does elsewhere.

Assuming Eric wasn’t lying in the original PR, direct requirements are needed to have pylint import checking work; so a 3rdparty requirement fetch for source code being linted is needed; although it looks like there has always been the bug that this requirement fetch is transitive instead of intransitive (again - assuming Eric’s commentary in the original PR is accurate).

We’re using black, flake8, mypy, isort (I would like to move to ruff). The idea that in my example, something would be using the dependency install mechanism for the actual linting of imports, is something that skipped my mind.

We have

flake8

et al. using the single

python-default

resolve / lockfiles to be clear.

If all you do is put

flake8

in your resolve/lockfile, this is not enough to use it as your linter. This only makes it available within your resolve. For example, if you wanted to

import flake8

in some of your source code at runtime. Even if you do not add flake8 to your resolve, adding the backend

pants.backend.python.lint.flake8

will get pants to enable flake8 as a linter. Pants ships with an internally pinned version of flake8, and will use this version unless you explicitly tell it to use the version from your lockfile.

I would’ve assumed Pants would discern that

flake8.pex

wouldn’t need most other dependencies itself as it does elsewhere.

For something like

flake8

it probably could, but this is a generic setting that is applied to a lot of tools. In the case of something like

pytest

, to which you can add a number of plugins (pytest-timeout for example, in my case), you can't actually determine these by dependency inference. (Pytest does a filesystem search to find plugins like this, but it never does

import <plugin>

so you need to know ahead of time that the plugins will be required) So the only safe default is to pull the whole lockfile. Using the

requirements

field allows pants to use a smaller subset and have better cache behavior.

> If all you do is put

flake8

in your resolve/lockfile, this is not enough to use it as your linter. This only makes it available within your resolve. For example, if you wanted to

import flake8

in some of your source code at runtime. Sorry no that’s not all we do, we have the relevant backends enabled also e.g.

pants.backend.python.lint.flake8

. It’s just that we control the version via one resolve

python-default

, rather than a different resolve. > Using the

requirements

field allows pants to use a smaller subset and have better cache behavior. For

flake8

specifically I would’ve thought this redundant, but it does seem to prevent seemingly unneeded dependencies from being installed on

pants lint

calling it.

For

flake8

specifically I would’ve thought this redundant, but it does seem to

I agree about the intuition, but It's well documented: pantsbuild.org/stable/reference/subsystems/flake8#… I think the issue is this generic code assumes the worst (cases like @freezing-wall-19707 describes for tools that support plugins out of the box that may not be in a custom resolve): github.com/pantsbuild/pants/blob/…/python_tool_base.py#… You can easily imagine that for other tools, the subclass could say - use the default requirements if using a custom lock and no requirements override is set by the user. I imagine if you care strongly enough about this intuition mismatch, you could argue for this feature and, if maintainers accept the idea, you could implement it fairly straightforwardly.

That makes sense. Thanks for the help folks, I’ll take a look at contributing this.