So let's say I have three python subprojects in a monorepo like this:

src/python/projectA
src/python/projectB
src/python/pipeline

and

projectB

is dependent on

projectA

. Both projects are using the

poetry_requirements

target and pyproject.toml files to specify their dependencies. Eventually

pipeline

will be deployed as a docker image, and has dependencies on both

projectB

and

projectA

. How do I indicate these internal dependencies to Pants to ensure that

projectB

and

projectA

sources are packaged and included in the docker image deployed from

pipeline

(or even when running the tests in

pipepline

)? It feels like I would maybe want to use parameterized resolves on each target within

projectA

and

projectB

?

From what I've seen, the intent isn't really to have separate projects that you intend to intermingle.

hmm I had some inkling that my organization mindset was not quite aligned with the goals of pants

Then when you go to package things out as distributions or however, Pants will take care of including what is necessary.

so for a python project, if one wanted to be really pants-thinking would you instead just have one big requirements.txt file for the whole repo, which Pants would just selectively grab dependencies from that are needed for building a specific distribution? and then if certain parts of the project needed a different version of a specific library you could override the dependency within the target spec for that part of the project?

Yup, that's exactly what it does. Only grabs the requirements it needs.

yeah I guess that's the part I'm worried about, I have some subprojects that need specific versions of specific libraries

yes a single resolve

its a little 🤯 but I think I'm getting it

I'll try to illustrate the complication I've found with multiple reoslves. Assume you have

A.py

,

B.py

, and

C.py

where

A

depends on

C

and

B

also depends on

C

. If

A

is supposed to come from one resolve and

B

comes from another resolve, then you have to also go parametize

C

so that Pants know which dependencies to pick up for

C

whether it's building the

A

package or the

B

package.

I think this is exactly the issue I'm running into now actually, I'm having to go down these parametrize rabbit holes every time a dependency from another resolve is added

If you must have projects have conflicting versions of a given requirement that is exactyl the point you bifurcate a resolve.

So we had multiple poetry projects like yours for a while and I was also trying to maintain multiple resolves etc but once you start having intra dependencies, which is expected in mono-repo, switching to some limited resolve is the only way. I also appreciate the fact that I don’t have to update same 3rd party packages multiple times in multiple poetry projects if I maintain a single or handful of resolves.

If you do it in the poetry way, I think you can use the

__defaults__

tooling to set up the parametrizing for you automatically.

There is a nice social aspect to a single resolve / a monorepo in general. You know what to expect at some base level as you move from project to project in your org and you can improve the whole org in 1 step by upgrading a dep with a security fix, etc.

yeah it's a very different way of thinking. my goal is to be able to support scientific developers in a way to enable code and tools sharing but also enable subprojects / analyses to have diverse sets (but often overlapping sets) of 3rd party dependencies. it seems like I may just have to evaluate projects one-by-one with regards to how to handle resolves (i.e. whether the project gets its own resolve or if it can be part of a larger context), but it's nice that Pants enables some flexibility there in terms of sharing or not sharing resolves on a case-by-case basis

Yeah, I think the main mental hurdle to make is that even though you're all "sharing" dependencies, there's only overlap when two things need the same package. If project A has 10 unique dependencies and project B has another unique 10, they won't be shared between each other at all.

for sure, and I see how that in itself might help with some of the dependency conflicts I've been running into in the past

So far, we haven't had many issues with that in our repo.

well I sure hope Pants will help relieve some of the issues we've been having in ours! so far it seems promising