The second issue is conceptual: We currently have one repository per library and build up pipeline stages by declaring the internal and external dependencies in

requirements.txt

. For some projects or some branches, we also pin internal dependencies. With a Pants-based monorepo, it makes sense to develop and test against the union of internal libraries, i.e., the source code as is. I can see how to cut different libraries from the monorepo as well. But I don't see a good solution to distributing libraries with pinned internal dependencies. Essentially, I'm asking how can I layer a packaged and possibly pinned view on top of the all sources in a big soup view of the monorepo with Pants. Any help in resolving this last remaining issue would be fantastic.

I'm not 100% sure I understand this, but have you looked at using generated setup.py files to create dists from your monorepo? There is logic there for overlaying a coarser-grained dist layout on top of the more fine-grained internal library layout, with each library being published by one owning dist. Is that in the neighborhood of what you're trying to achieve?

@happy-kitchen-89482, I read that a few weeks ago. The way I understood it is that it covers how to create individual distributions. The problem I'm trying to solve is the graph of distributions. Let's say we have to libraries A and B, both created from the monorepo and A depends on B. How can I pin B at a particular release? Also, lets assume there is another library C that also depends on B. But I'd like to pin C's B to a different version. Also, D may just prefer the latest B and not require pinning. I'm not clear on how to do that, in particular how to capture those extra version constraints.

Hey sorry, was traveling and this slipped below the fold, thanks for the reminder!

Thank you, @happy-kitchen-89482. That was super-helpful. You just helped me realize that I've been making the problem more complex than it need be — in the context of our existing practices: First, even the possibility of a monorepo on the horizon has already started an effort towards eliminating unnecessary internal variability from the code base. There are about 10 or so internal libraries that are re-used across most data processing pipelines. Any pins have rapidly been vanishing. Yay! Second, there is one version of those pipelines that must evolve at a much slower speed. We currently use branches for the purpose. But I just realized that nothing about Pants prevents us from continuing to use a dedicated branch. In that branch, we can do all the pinning we want, but it needn't spill over into the main branch. In short, nothing really changes besides that we don't deal with a repo per library anymore, but slowly try to converge on one repo. Thank you!!!

@freezing-photographer-88553 I think this is a similar problem to the one we discussed earlier, and are converging on a similar solution!