Hi there, I’m sorry if I’m repeating a question that you might have received and answered already, I wasn’t able to find any history though. I’m also sorry for this wall of text 🙃. I would like to have your opinion on some matter involving multiple resolves for Python in Pants. I’m looking for an ideal way to manage multiple resolves in a monorepo that might just need to diverge in a minimum way. As a use case, I can report on something that is happening in the monorepo I’m working on, which is day by day increasing in the number of projects and tightening the requirements. There are many projects that are shared, mostly of them as libs. However, sometimes there are projects that depend on those libs but will never be imported from others. And here it comes the situation in which sometimes we face problems. A new project might have to include a new third-party dependency which is actually incompatible with the default resolve. Additionally, having a trunk-based development, there might be developers wanting to add a new dependency even if temporarily, but that will have to be included in the default resolve anyways. They might want to use a different resolve, but that doesn’t break dependencies with shared projects. I wanted to describe two use cases just to give more reasoning to the solutions I was thinking. Ideally, I see a set of requirements that is global and for sure referring to all those projects that might become dependencies for others (like a libs resolve). However, I would like to have additional resolves that inherit all those global requirements, but extending some of them. One solution might be to have a global requirements.txt file defining a resolve. Then, a new resolve would be defined by including the global one (I think is possible by using the target for Python requirements). However, Pants would build the extended one with a new lockfile and probably not pinning the same versions of the global one. So, I would say that the global lockfile should act as a sort of list of constraints of versions to be considered when generating the second. Do you think this could be easily achieved with Pants? Would it maybe require to work with the contents of the lockfiles with some scripts?

Not to ping-pong this back, but just to clarify a few items that I'm struggling to grok from your message (not necessarily related to what you wrote - I haven't had my coffee yet 🤦‍♂️). > an ideal way to manage multiple resolves in a monorepo that might just need to diverge in a minimum way So are you using multiple resolves or a single resolve? Because from your message, I thought you had a multitude of resolves, but you want a single resolve? > However, sometimes there are projects that depend on those libs but will never be imported from others Not that it necessarily solves your deps problem, but are you using visibility rules to enforce this? Or just convention, or something else? > A new project might have to include a new third-party dependency which is actually incompatible with the default resolve So, what is the current action when this happens? > there might be developers wanting to add a new dependency even if temporarily, but that will have to be included in the default resolve anyways. They might want to use a different resolve, but that doesn’t break dependencies with shared projects. Adding a temporary dependency, included in default but might want to use a different resolve? I don't think I understand this workflow. > However, I would like to have additional resolves that inherit all those global requirements, but extending some of them. > ... > So, I would say that the global lockfile should act as a sort of list of constraints of versions to be considered when generating the second I think this is the meat of what you'd like, right? A set of defaults that are conditionally overridden? Or extended? What are you doing today to mitigate these problems? What is the existing problem with each project having it's own resolve?

By re-reading it I admit I'm having some struggles too 😄 I'm sorry, I'll try to explain a bit better.

I'm also guessing there is a problem doing something liek this? https://www.pantsbuild.org/2.18/docs/python/overview/lockfiles#multiple-lockfiles

We have a monorepo, and we are religiously keeping one single resolve. We have a set of (mostly)

libs

that are shared and some

projects

that use them. Everything works well and at the current status all matching the requirements. I'm explicitly distinguishing between libs (shared) and projects. However, there are two cases we have to deal with, which might break things. 1 - A developer is working on a new project and its project being in a

development

status. They would like to continue to integrate all the transitive dependencies compatible with the main resolve, however they also want to have some freedom like in a sandbox. 2 - We have a project that will make use of some of the

libs

, potentially never being a dependency for other project, and introducing a third-party dep that creates some conflicts for whatever reason with the set currently in the resolve. In both situations, we are currently forcing to make it work. For instance, if that library has to enter into the requirements and fails for some strict or misconfigured

setup.py

, then we fork the project and relax those strict requirements to build a modified wheel. The same happens in case (1), when the developers are still unsure whether that third-party dependency will ever be adopted, but in the meantime struggling to make it work with the monorepo. This is not effortless, and happening more often. Therefore, I'm trying to identify a way to manage this. I see that there is a minimum (which is actually big) of deps that has to be global. I would identify this with the

libs

resolve. Then, I would like to have any other projects being part of other resolves (even one each), but all inheriting the same set of dependencies and pinned versions of the third-party deps for

libs

. I don't know if made my self clear better this time 😕

Yep, this makes more sense

We have a monorepo, and we are religiously keeping one single resolve.

.... Whelp, I see a problem 🙂 What's the reason for this? It sounds pretty explicitly like sticking to a single-resolve is the underlying problem, no? And, so because everything needs to be in a single resolve - something like this won't fly?

python_requirements(
    name="webapps_shared",
    source="webapps-shared-requirements.txt",
    resolve=parametrize("webapps_django3", "webapps_django4")
)

I guess I'm not understanding how what you were proposing above would solve the problem with transitive dependencies. Like, you have a large resolve file, but then want to override a dep - that would have to transitively override dependencies all the way down, no? So aren't you just at multiple resolves with more steps? Is there any way to make relaxed constraints files work? https://www.pantsbuild.org/2.18/reference/subsystems/python#resolves_to_constraints_file

The reason why we want to have a single resolve is to have, at least for the

libs

consistency with the third-party libraries with exact versions (security policies, for instance). At this point let's call it as

libs

. It's not really that I have a big resolve that I want to override. It's more that I want to extend that by adding something for a single project. If I simply replicate the

libs-requirements.txt

file, how can I guarantee to pick the same versions of the common libraries (in

libs

lockfile) for all the extended resolves?

Do you happen to have a stripped down repo to give a concrete example of this problem? Again, I'm sure I'm misunderstanding something here - but I thought some of the Pants per-project resolves could maybe handle this? Like, use your global resolve the majority of the time, and then either add another resolve on certain projects, or maybe even use

overrides

I don't have a repo at the moment but I could think about creating one asap. However, to shorten if it helps at the moment: I have a resolve called

libs

referring to Python project that are supposed to be imported. They have a

requirements-libs.txt

file, let's say there is

requests>1.0.0

. I generate a

libs.lockfile

that pins the version of the third party libraries, which pick

requests==1.0.1

. Then, I have projects

A

and

B

, about a problem that we actually have. They are both using some of the

libs

.

A

adds

batchgenerators

in the requirements list, which has the transitive requirement of

unittest2

.

B

instead wants to use

dbt

which is incompatible with

unittest2

. To be more precise, just having

unittest2

makes it fails when launched. In this situation

libs

cannot work as a global resolve. What I would like to do is to create for each of them a new resolve and their own

requirements.txt

file, but by inheriting the requirements and exact versions from

libs

. If I simply tell the target to use

requirements-libs.txt

, I don't think I can guarantee to end up with the exact same version in

libs.lockfile

, like

requests==1.0.1

. I need those exact version to be used as constraints when generating

A.lockfile

and

B.lockfile

, essentially.

I think I'm following the use case. By the exact versions, you mean the "default to the versions in the shared lock file but make any changes to handle the compatibilities that lead me to create a new resolve in the first place". And just for context, is something you had a satisfactory workflow for with pip/pex/poetry/conda/some-other-tool?

I would treat the shared lockfile as constraints an not to be changed at all. If any dependency is in there, it will stay and keep the versions once the lockfile is generated. Project A and B will need to be strictly compatible with the requirements in libs, but I want them to allow the addition of new requirements independently. This because A and B will be independent when deployed and never have to depend on each other. This means that the developers of A will have to do the effort to have a set of requirements compatible with libs, but they shouldn’t care if they will have some conflicts with B. Unfortunately, I don’t have history where I used a different tool to solve this approach. At the moment I see a couple of possible approaches deriving from this discussion. I create a libs resolve with a related libs-requirements.txt file. It will be a reference for all the libs, the shared projects, in the monorepo. I therefore create the lockfile. Now, if I add a project I see two options. I refer to the libs-requirements.txt file plus a custom requirements.txt. With the new generation, it might be that some of the versions contained in libs.lockfiles will be different from the new generated ones in terms of of which version has been resolved. This combination should be easy to achieve with pants. I have to consider pros and cons of this solution yet. Other option: I read the libs.lockfile and produce libs-constraints.txt file. Therefore, when defining the new resolve for the new project, I will have a combination of libs-requirements.txt + requirements.txt + constraints.txt. In this way I should have the same versions mentioned in libs.lockfile and the new additions.

So if you could take lockfile

foo

,

cp

it and then add new projects to it without changing the existing pins would that help?

I think so! If there is a way convenient for the developers also…

https://github.com/pantsbuild/pants/issues/15704 is along that lines and I'm hopeful will see some movement soon. That would still leave you with a somewhat rough "copy the lockfile and do stuff to the fork" workflow, and I'm not sure about keeping the "forks" in sync as the "libs" change.

Thank you! I’ll have a look 👀. At the moment we have about 165 requirements 😑