https://pantsbuild.org/ logo
#general
Title
# general
r

rough-vase-83553

09/30/2022, 2:16 PM
Question about the use of
python_requirements
. I'm aware that you can create an arbitrarily large lockfile without having to worry about everything in the lockfile getting dragged in to each Python target. But do you have to also create separate
python_requirements
for each binary / library if you want to restrict the number of dependencies they pull in? Or can Pants still resolve which ones to pull in via import statements?
1
e

enough-analyst-54434

09/30/2022, 2:24 PM
By
python_dependencies
do you mean
python_sources.dependencies
? Perhaps you mean
python_requirements
? Either way, when building a venv / Pex for a given target, Pants will always only include the subset of deps it (transitively) needs. That information will come from a combination of dependency inference (analyzing imports as you guessed) and explicit dependecies lists on various `python_X`targets that need coaching.
🙌 1
r

rough-vase-83553

09/30/2022, 2:25 PM
Whoops sorry
python_requirements
e

enough-analyst-54434

09/30/2022, 2:25 PM
Yeah, ok - so no need to use more than one of those unless you have resolve conflicts and are forced into multiple resolve territory. Even then though, the same minimalism applies.
r

rough-vase-83553

09/30/2022, 2:27 PM
Oh so I guess concretely my question is this. If you specify boto3>1.17.0 in the
python_requirements
and apply it to everything in a directory tree, will every target in the directory tree pull in boto3 even if they don't really need it?
i.e. Should you generally avoid specifying any
python_requirements
at and expect Pants to resolve deps intelligently?
e

enough-analyst-54434

09/30/2022, 2:36 PM
No. So perhaps best to summarize like this: whatever `./pants dependencies <file or target>`shows is what will be included in the venv for that file or target - nothing more. Dependencies are calculated only from mapping import statements to 1st party or 3rdparty modules. You supplement misses there with manual
dependecies
list additions. In either case, if neither a file or target (or its transitive dependencies)
import boto3
, then it will not get a dep on boto3 and, as a result, it will not get boto3 included in its venv / PEX.
👍 2
So, to the contrary,
python_requirements
should include all direct dependencies of all code in your repo.
At some point, as your repo grows, there will be a dependency conflict. Only at that point do you need to introduce a new `python_requirements`/ resolve.
r

rough-vase-83553

09/30/2022, 2:39 PM
Gotcha. And what's the right way to specify which
python_requirements
applies to a project directory if not for
sources.dependencies
? I think I've been putting
python_requirements
in the
sources.dependencies
for everything to avoid ambiguity since we have some lingering requirements files that are getting turned into
python_requirements
by tailor
But it seems like that has the effect of bloating the true resolved deps
e

enough-analyst-54434

09/30/2022, 2:40 PM
Yeah, any dependency you manually add to dependencies lists will turn off Pants smarts. You said to add it and Pants won't second guess you. You asked for the bloat.
2
So, with multiple requirements files /
python_requirements
targets you're likely in multiple resolve territory unless I'm not seeing some alternate setup. Do you have 1 lock file per requirements file right now?
r

rough-vase-83553

09/30/2022, 4:29 PM
Yeah I think the best strategy is to probably prevent Tailor from generating
python_requirements
for these lingering requirement files or manually remove them.
Although do you think it would make more sense to have requirements specifiable at the Python target level?
h

hundreds-father-404

09/30/2022, 4:31 PM
to have requirements specifiable at the Python target level?
They can be, via manually creating
python_requirement
targets.
python_requirements
is a "target generator" that creates one
python_requirement
target for you: it solely exists for boilerplate reduction
🤔 1
r

rough-vase-83553

09/30/2022, 4:32 PM
So does it interact with e.g. every
python_library
target in the subdirectory, every
python_library
associated with the same
resolve
?
h

hundreds-father-404

09/30/2022, 4:35 PM
Yeah: the first section of this https://www.pantsbuild.org/docs/python-third-party-dependencies explains the mental model. There is a "universe" of valid
python_requirement
targets your code can use. Usually, your code uses a subset of those, which Pants figures out via dependency inference + your explicit deps Those
python_requirement
targets are defined by you, and you can use target generators like
python_requirements
for less boilerplate If you are using "multiple resolves" (aka lockfiles), you may have multiple "universes". Each "universe" is self-contained, and code can only choose exactly one resolve/universe
r

rough-vase-83553

09/30/2022, 4:47 PM
Ah okay it happens purely through resolve then. Hmm...
5 Views