#19376 Support specifying `project.optional-dependencies` extras on the `python_requirements` target. Issue created by AlirezaRoshanzamir Problem In Pants 2.16, there is a requirement to specify a lock file for each tool. Currently, this specification is not mandatory, but the requirements field for each tool does not support additional dependencies, such as

flake8-annotations

for

flake8

or

pyenchant

for

pylint

. This limitation exists because of Pants' internal lock files. Additionally, Pants enforces the use of Pants-generated lock files for the tools. While Poetry and Pipenv support grouping requirements, they strangely do not support generating multiple lock files for each group. As a result, when working with our project's repository, we end up with multiple requirements files and corresponding lock files, totaling around 20 files. We decided to address this by placing all the requirements in the

pyproject.toml

file using the PEP 621 format and organizing each tool's requirements in a separate group inside the

project.optional-dependencies

section. However, since there are different lock files for each group, we cannot specify this in the

python_requirements

target. Proposed solution To address this issue, we propose enhancing the

python_requirements

target to allow the selection of a subset of the

pyproject.toml

file. Currently, Pants understands optional dependencies and generates a tag for each one. Our suggestion is to add the capability to specify a specific part of the

pyproject.toml

file in the

python_requirements

target:

python_requirements(
    name="lint",
    source="pyproject.toml",
    extra="lint",
    resolve="lint",
)

However, we acknowledge that this idea conflicts with the installation of requirements using Pip and

pyproject.toml

. According to PEP 621 and similar to the

extra_requires

concept in the

setuptools

literature, these optional dependencies should be installed in addition to the main requirements, rather than being separate entities. Perhaps we should consider avoiding the use of the

extra

keyword and instead find a more generic approach to solve this issue:

python_requirements(
    name="lint",
    source="pyproject.toml",
    variable="[project.optional-dependencies].lint",
    resolve="lint",
)

Alternative approaches attempted We have explored alternative lock file generators such as Poetry and Pipenv. While both tools support grouping requirements, they do not support the generation of multiple lock files. Furthermore, Poetry enforces that all requirement groups should be compatible with each other, even if they are optional. These limitations have been discussed in the following GitHub issues: link 1, link 2, and link 3. In addition to the aforementioned problems, Pants mandates the use of Pants-generated lock files for the tools, further complicating the situation. Despite our attempts, using a single

requirements.txt

and

requirements.lock

file led to conflicts with the

importlib-metadata

library when incorporating new versions of

flake8

and

sphinx

. Finally, I would greatly appreciate any alternative solutions that can help avoid the proliferation of multiple requirements files and lock files. pantsbuild/pants