Q11 Making a dependency to a `requirements txt` using `pytho

Question

Q11 Making a dependency to a `requirements txt` using `python requirements ` seems to put all the same external dependencies to different packages defined as `python distribution ` in multiple different `BUILD` files Through my Q5 A5 I expected that pants automatically include the actual direct dependencies only but it seems not doing any filtering like that How do I achieve it Example pkg manager depends on aiofiles but pkg common does not They both depends on aiohttp Both aiofiles and aiohttp are included in the unified `requirements txt` manager depends on common I just removed `pyproject toml` and let pants generate setup py using `generate setup=True` gt expected `METADATA` of the manager pkg include aiofiles while the common pkg s does not Both include aiohttp because although manager depends on common only the direct requirements of manager should be added to manager s pkg metadata not both manager and common s direct requirements gt actual result Both include aiofiles and aiohttp

happy-kitchen-89482 · Answer

Pants should only include dependencies that are actually needed unless you add them explicitly in the `dependencies=` part of a target

happy-kitchen-89482 · Answer

So this is odd

happy-kitchen-89482 · Answer

The issue is that common s METADATA includes `aiofiles` but it shouldn t

happy-kitchen-89482 · Answer

it sounds like

rhythmic-morning-87313 · Answer

yes that s the issue

happy-kitchen-89482 · Answer

and common doesn t import from aiofiles

rhythmic-morning-87313 · Answer

it does not

happy-kitchen-89482 · Answer

and you haven t added any explicit dependencies either

rhythmic-morning-87313 · Answer

rhythmic-morning-87313 · Answer

you could test with this PR branch

rhythmic-morning-87313 · Answer

` pants package src ai backend common dist` and ` pants package src ai backend manager dist` unzip `dist whl` and inspect the package s METADATA from ` egg info` directories

rhythmic-morning-87313 · Answer

I think pants is just blindly copying all the requirements because `src ai backend manager service` depends on ` reqs` directly

rhythmic-morning-87313 · Answer

< curved television 6568> has mentioned the new resolves feature along with A5 so I think I m not using it properly

happy-kitchen-89482 · Answer

Ah did you add an explicit dependency on ` reqs` That would explain things

happy-kitchen-89482 · Answer

Pants will never remove an explicit dependency

happy-kitchen-89482 · Answer

But if you omit that dependency and just let Pants infer the dependencies that are actually needed the right thing should happen

happy-kitchen-89482 · Answer

` reqs` generates a separate target for each requirement in the file but if you depend on the generator itself you re telling Pants to depend on everything it generates

rhythmic-morning-87313 · Answer

oh yes removing ` reqs` dependency from `python sources ` makes the correct wheel

rhythmic-morning-87313 · Answer

i ve put it because running tests had missing 3rd party dependencies

rhythmic-morning-87313 · Answer

but it also works now

rhythmic-morning-87313 · Answer

so implicitly `python requirements ` is recognized by pants

rhythmic-morning-87313 · Answer

great thanks

happy-kitchen-89482 · Answer

98 of the time Pants will figure out the `dependencies` for you mostly based on `import` statements If you do need to add a specific one that couldn t be inferred from static analysis you can

happy-kitchen-89482 · Answer

But even then you d add just the specific thing you need ` reqs ansicolors` for example

happy-kitchen-89482 · Answer

Not all of ` reqs`

rhythmic-morning-87313 · Answer

good to know that

rhythmic-morning-87313 · Answer

i have another following question

happy-kitchen-89482 · Answer

We should make this more clear in the documentation

rhythmic-morning-87313 · Answer

is it possible to check any missing entry in `requirments txt` e g those are imported in the code but requirements does not have it yet

rhythmic-morning-87313 · Answer

this may be tricky because package names and import names may differ and the docs is already mentioning custom mapping of it

rhythmic-morning-87313 · Answer

if a package X exposes an import y then how does pants handle it when 1 filtering only actually used requirements as in my case 2 figuring out missing requirements

rhythmic-morning-87313 · Answer

since you ve just mentioned it is doing a static analysis

rhythmic-morning-87313 · Answer

should i manually put such mappings

rhythmic-morning-87313 · Answer

a typical example that comes in my mind is `attrs` exposing `attr` and `attrs`

happy-kitchen-89482 · Answer

Generally if pants sees `from foo import bar` it will look for something that provides `foo bar` and that something can be either code in your repo for example `src foo bar py` or a third party requirement Often that will be called `foo` on PyPI but if it s not say it s called `foolib` then you can add that mapping in the `python requirements` target in the BUILD file <https www pantsbuild org docs reference python requirements codemodule mappingcode>

happy-kitchen-89482 · Answer

Pants ships with some default module mappings and we re building that list up over time

happy-kitchen-89482 · Answer

But if nothing matches that import then today Pants doesn t do anything with that information

happy-kitchen-89482 · Answer

It could though But usually people find out about missing deps because tests fail

happy-kitchen-89482 · Answer

Sorry I m wrong you can get Pants to log a warning for these unowned dependencies

happy-kitchen-89482 · Answer

This does that for Python <https www pantsbuild org docs reference python infer section unowned dependency behavior>

happy-kitchen-89482 · Answer

I don t think we support this for other languages maybe we should

rhythmic-morning-87313 · Answer

i also thoguht missing dependencies would fail in tests checks but knowing about the unowned dependency handling option is useful as well