https://pantsbuild.org/ logo
#general
Title
# general
s

swift-river-73520

03/24/2023, 4:35 PM
So let's say I have three python subprojects in a monorepo like this:
Copy code
src/python/projectA
src/python/projectB
src/python/pipeline
and
projectB
is dependent on
projectA
. Both projects are using the
poetry_requirements
target and pyproject.toml files to specify their dependencies. Eventually
pipeline
will be deployed as a docker image, and has dependencies on both
projectB
and
projectA
. How do I indicate these internal dependencies to Pants to ensure that
projectB
and
projectA
sources are packaged and included in the docker image deployed from
pipeline
(or even when running the tests in
pipepline
)? It feels like I would maybe want to use parameterized resolves on each target within
projectA
and
projectB
?
h

high-yak-85899

03/24/2023, 4:36 PM
From what I've seen, the intent isn't really to have separate projects that you intend to intermingle.
Rather, if you have separate projects, then they stay truly separated.
While you can organizationally think about your code as separate projects, things are much easier if they all share common requirements and such when possible.
s

swift-river-73520

03/24/2023, 4:37 PM
hmm I had some inkling that my organization mindset was not quite aligned with the goals of pants
h

high-yak-85899

03/24/2023, 4:37 PM
Then when you go to package things out as distributions or however, Pants will take care of including what is necessary.
Since poetry doesn't do dependency graphing, their solution is to segment your projects by folders and such.
Then you tell poetry how those folders are shared together.
But that's not really a thing in Pants. So the benefit of separating projects is more for humans than the build tooling.
s

swift-river-73520

03/24/2023, 4:48 PM
so for a python project, if one wanted to be really pants-thinking would you instead just have one big requirements.txt file for the whole repo, which Pants would just selectively grab dependencies from that are needed for building a specific distribution? and then if certain parts of the project needed a different version of a specific library you could override the dependency within the target spec for that part of the project?
h

high-yak-85899

03/24/2023, 4:49 PM
Yup, that's exactly what it does. Only grabs the requirements it needs.
Idk about overriding though
We just maintain one big file
One downside that's been causing is that generating lockfiles takes longer and longer the bigger that file gets
s

swift-river-73520

03/24/2023, 4:50 PM
yeah I guess that's the part I'm worried about, I have some subprojects that need specific versions of specific libraries
are you just using a single resolve?
I guess with this style of thinking you would err on the side of less resolves or a single resolve
h

high-yak-85899

03/24/2023, 4:51 PM
yes a single resolve
s

swift-river-73520

03/24/2023, 4:51 PM
its a little 🤯 but I think I'm getting it
h

high-yak-85899

03/24/2023, 4:53 PM
I'll try to illustrate the complication I've found with multiple reoslves. Assume you have
A.py
,
B.py
, and
C.py
where
A
depends on
C
and
B
also depends on
C
. If
A
is supposed to come from one resolve and
B
comes from another resolve, then you have to also go parametize
C
so that Pants know which dependencies to pick up for
C
whether it's building the
A
package or the
B
package.
That probably wasn't very clear and I'm sure I'm not aware of all the proper techniques to handle this, but we've found things are far easier if you can just get everyone to agree on the requirements to use
And, for the scenarios you can't, hopefully you don't have to share a lot of code between the two projects.
s

swift-river-73520

03/24/2023, 4:54 PM
I think this is exactly the issue I'm running into now actually, I'm having to go down these parametrize rabbit holes every time a dependency from another resolve is added
this has massively improved my understanding of the options and value proposition though, thank you much for the explanation / tips
e

enough-analyst-54434

03/24/2023, 4:57 PM
If you must have projects have conflicting versions of a given requirement that is exactyl the point you bifurcate a resolve.
Hopefully there are very few instances of this. A handful, 2-3 at most.
r

refined-addition-53644

03/24/2023, 4:59 PM
So we had multiple poetry projects like yours for a while and I was also trying to maintain multiple resolves etc but once you start having intra dependencies, which is expected in mono-repo, switching to some limited resolve is the only way. I also appreciate the fact that I don’t have to update same 3rd party packages multiple times in multiple poetry projects if I maintain a single or handful of resolves.
h

high-yak-85899

03/24/2023, 5:00 PM
If you do it in the poetry way, I think you can use the
__defaults__
tooling to set up the parametrizing for you automatically.
But I think that forces the developer to put a lot of thought into repo layout and walk very fine lines or else things explode on them.
e

enough-analyst-54434

03/24/2023, 5:03 PM
There is a nice social aspect to a single resolve / a monorepo in general. You know what to expect at some base level as you move from project to project in your org and you can improve the whole org in 1 step by upgrading a dep with a security fix, etc.
💯 1
That said, I've definitely seen plenty of folks / teams absolutely not like this. They prefer an isolated sandbox for their project code.
Definitely wars have been waged!
⚔️ 1
s

swift-river-73520

03/24/2023, 6:10 PM
yeah it's a very different way of thinking. my goal is to be able to support scientific developers in a way to enable code and tools sharing but also enable subprojects / analyses to have diverse sets (but often overlapping sets) of 3rd party dependencies. it seems like I may just have to evaluate projects one-by-one with regards to how to handle resolves (i.e. whether the project gets its own resolve or if it can be part of a larger context), but it's nice that Pants enables some flexibility there in terms of sharing or not sharing resolves on a case-by-case basis
h

high-yak-85899

03/24/2023, 6:12 PM
Yeah, I think the main mental hurdle to make is that even though you're all "sharing" dependencies, there's only overlap when two things need the same package. If project A has 10 unique dependencies and project B has another unique 10, they won't be shared between each other at all.
Of course, transitive dependencies can cause issues. Even though the direct dependencies are unique, they may be transitively linked and cause some communication to happen to fix that.
s

swift-river-73520

03/24/2023, 6:13 PM
for sure, and I see how that in itself might help with some of the dependency conflicts I've been running into in the past
h

high-yak-85899

03/24/2023, 6:13 PM
So far, we haven't had many issues with that in our repo.
s

swift-river-73520

03/24/2023, 6:16 PM
well I sure hope Pants will help relieve some of the issues we've been having in ours! so far it seems promising
definitely one of those standing on the shoulders of giants moments though
2 Views