https://pantsbuild.org/ logo
Join Slack
Powered by
Q11: Making a dependency to a `requirements.txt` u...
# general
r
Q11: Making a dependency to a 
requirements.txt
using 
python_requirements()
seems to put all the same external dependencies to different packages defined as 
python_distribution()
in multiple different 
BUILD
files. Through my Q5/A5, I expected that pants automatically include the actual direct dependencies only, but it seems not doing any filtering like that. How do I achieve it? Example: pkg "manager" depends on "aiofiles" but pkg "common" does not. They both depends on "aiohttp". Both "aiofiles" and "aiohttp" are included in the unified 
requirements.txt
. "manager" depends on "common". I just removed 
pyproject.toml
and let pants generate setup.py using 
generate_setup=True
. -> expected: 
METADATA
of the "manager" pkg include "aiofiles" while the "common" pkg's does not. Both include "aiohttp". (because although "manager" depends on "common", only the direct requirements of "manager" should be added to "manager"'s pkg metadata, not both "manager" and "common"'s direct requirements) -> actual result: Both include "aiofiles" and "aiohttp".
✅ 1
h
Pants should only include dependencies that are actually needed, unless you add them explicitly in the 
dependencies=
part of a target.
So this is odd
The issue is that common's METADATA includes 
aiofiles
but it shouldn't
it sounds like
r
yes, that's the issue
h
and common doesn't import from aiofiles?
r
it does not
h
and you haven't added any explicit dependencies either?
r
you could test with this PR branch
./pants package src/ai/backend/common:dist
and 
./pants package src/ai/backend/manager:dist
, unzip 
dist/*.whl
, and inspect the package's METADATA from 
*.egg_info
directories
I think pants is just blindly? copying all the requirements because 
src/ai/backend/manager:service
depends on 
//:reqs
directly.
@curved-television-6568 has mentioned "the new resolves" feature along with A5, so I think I'm not using it properly...
h
Ah, did you add an explicit dependency on 
//:reqs
? That would explain things
Pants will never remove an explicit dependency
But if you omit that dependency and just let Pants infer the dependencies that are actually needed, the right thing should happen
(
//:reqs
generates a separate target for each requirement in the file, but if you depend on the generator itself, you're telling Pants to depend on everything it generates)
r
oh, yes, removing 
//:reqs
dependency from 
python_sources()
makes the correct wheel!
i've put it because running tests had missing 3rd party dependencies
but it also works now!
so implicitly 
python_requirements()
is recognized by pants
great, thanks!!
h
98% of the time Pants will figure out the 
dependencies
for you, mostly based on 
import
statements. If you do need to add a specific one that couldn't be inferred from static analysis you can
🙌 1
But even then, you'd add just the specific thing you need. 
//:reqs#ansicolors
for example
Not all of 
//:reqs
r
good to know that
i have another following question
h
We should make this more clear in the documentation...
r
is it possible to check any missing entry in 
requirments.txt
? (e.g., those are imported in the code but requirements does not have it yet)
this may be tricky because package names and import names may differ (and the docs is already mentioning custom mapping of it)
if a package "X" exposes an import "y", then how does pants handle it when 1) filtering only actually used requirements as in my case, 2) figuring out missing requirements?
since you've just mentioned it is doing a static analysis
should i manually put such mappings?
(a typical example that comes in my mind is 
attrs
-- exposing 
attr
and 
attrs
)
h
Generally, if pants sees 
from  foo import bar
it will look for something that provides 
foo.bar
, and that something can be either code in your repo (for example 
src/foo/bar.py
) or a third party requirement. Often that will be called 
foo
on PyPI, but if it's not (say it's called 
foolib
) then you can add that mapping in the 
python_requirements
target in the BUILD file: https://www.pantsbuild.org/docs/reference-python_requirements#codemodule_mappingcode
Pants ships with some default module mappings, and we're building that list up over time
But if nothing matches that import then today Pants doesn't do anything with that information
It could though! But usually people find out about missing deps because tests fail.
👍 1
Sorry, I'm wrong, you can get Pants to log a warning for these unowned dependencies!
I don't think we support this for other languages, maybe we should
r
i also thoguht missing dependencies would fail in tests/checks, but knowing about the unowned dependency handling option is useful as well!
5 Views
Open in Slack
PreviousNext
Powered by