Hi, I’m really curious how people here manage dependencies in pants, especially when adding a new dependency into the universe. For example, in a plain python environment, I might do the following

> pip install boto3-stubs[ec2]
> pip freeze
boto3-stubs==1.28.85
botocore-stubs==1.31.85
mypy-boto3-ec2==1.28.85
types-awscrt==0.19.10
types-s3transfer==0.7.0
typing_extensions==4.8.0

In the pants world if I have a requirements.txt with just

boto3-stubs[ec2]

, then I would get a warning that there’s no owner if I use the transitive import like:

from mypy_boto3_ec2 import Client as Ec2Client

My goals: 1. I would like to not get any warnings when running pants commands 2. I would like to have a simple way of adding new dependencies. e.g. just adding

boto3-stubs[ec2]

and not all the dependencies It’s pretty bad developer experience if in order to pull a dependency, I have to first manually figure out the transitive dependencies and then merge it into the requirements.txt. I’m curious how do people solve this problem here.

as soon as you import it in your code, it’s no longer a transitive dependency but a direct one. With that said, if it comes as a kind of direct result of some parent umbrella distribution, I think using the module mappings to tell pants which of your third party dependencies provide that module would be the way to go.

It’s pretty bad developer experience if in order to pull a dependency, I have to first manually figure out the transitive dependencies and then merge it into the requirements.txt.

I think this is actually the other way around: it's a bad developer experience if a third-party library changing their dependencies suddenly removed it from your project. Terrible to debug, because just regenerating lockfiles without any code changes will suddenly make CI go red! This has happened a few times for us when using "simpler" solutions like PDM or Poetry, and much worse when we had our own custom system. Andreas does raise a valid point regarding collection-packages, but I think the extra pain in those situations is worth the stronger guarantees in other situations.

Thanks Andreas and Tom. I agree on a few points around correctness and safety respectively: 1. “as soon as you import it in your code, it’s no longer a transitive dependency but a direct one.” In this regard, I totally agree it should be surfaced in the requirements somehow. I personally very much like pip-tool’s pip-compile to generate a lockfile to show explicitly what is used. 2. “just regenerating lockfiles without any code changes will suddenly make CI go red”. I also totally agree this should not be the case. Maybe this is related to the next message but if upstream shipped broken library (missing deps) it’s kind of on them. A lock file would definitely help mitigate this. If I’m understanding correctly, the suggested workflow for adding new dependency is the following: a. When adding a new library, add it directly to requirements.txt (or equivalent) so that pants make this a python_source target. b. If the code only imports this library directly, then the job is done. c. If in code depends on a transitive library (in the example above

boto3-stubs

is the pip package, but the main usage is via

mypy-boto3-ec2

then it should be added directly into requirements.txt as well I do think that the above is still pretty manual and puts a lot of onus on the developer. e.g. as a developer, when I put

boto3-stubs

in requirements.txt, I should generate lock file, and then when I start writing code and import ``mypy-boto3-ec2` which I know comes from

boto3-stubs

I need to go add it in requirements.txt and then generate the lock file again. When I started this thread, I was very curious how people add dependencies. Do they do what I’ve just mentioned above, or is there some other workflow that people use which is more ergonomic.

Yea, pretty much the above. Which is why I’ve filed this feature request (a while back) to make this more ergonomic: https://github.com/pantsbuild/pants/issues/12880 At the time this was in the midst of implementing much of the lockfile infrastructure.. but now that has settled this should be able to be addressed, I think..

Oh yes, that was a typo

isn’t that the lockfile, pretty much..?

I mean the ergonomics of adding something to the lockfile.

oh.. does

go get <dep>

add to the list of deps. in

go.mod

?

Yup. So in the above, if I ran

go get foo

and foo depends on bar, and dep2 v2 Then I believe

go.mod

will become

dep // direct
dep1 // indirect
dep2 (v2 upgraded) // indirect
foo // direct (assume usage in real code)
bar // indirect

that could help here, in case of the collection-package scenario..