#19658 Monorepo setup using Poetry for different classes of projects New discussion created by zbitouzakaria Hi everyone! I'm working on setting up a monorepo using Pants and Poetry to manage multiple classes of Python projects, and I would love to gather your feedback on the choices made. We have specific requirements to factor in: Requirements and Intent We have two main classes of projects: •

release

projects: These are the ones that have deployed artifacts running on our cloud. We want to ease the development of new services by having them neatly organized in a single repo, allowing easy composition of libs for shared usage. We can use visibility rules to ensure a service cannot depend on another service and these can be subdivided in multiple "universes" of dependencies, through resolves. Classic

Pants

use case. •

sandbox

projects: These are of the exploratory kind (e.g some data science projects). We want them isolated since they might use packages with unstable APIs, or just fill a quick need for insights. We don't put them in a separate repo because we believe we can ease the bootstrap of such projects as they might need specific cloud infrastructure, and they could benefit from using libs from our

release-*

resolves. Usefulness will be limited if they have to comply to our whole universe of dependencies, so they need to be able to pick up just a lib and it's direct dependencies. I started from poetry#6850 and worked from there. Monorepo Structure

.
├── lockfiles
│   └── pytest.lock
├── tf
│   ├── modules/ # Common terraform modules
├── py
│   ├── sandbox/
│   │   ├── projects/ # Classic poetry projects
│   │   │   ├── [...]
│   ├── release-core/ # Core resolve
│   │   ├── .venv/ # Poetry virtualenv for the resolve, can be useful for development
│   │   ├── 3rdparty
│   │   │   └── release-core.lock
│   │   ├── libs
│   │   │   └── lib1
│   │   │       ├── org-namespace
│   │   │       │   └── lib/
│   │   │       ├── BUILD
│   │   │       └── pyproject.toml
│   │   ├── projects
│   │   │   ├── app
│   │   │   │   ├── org-namespace
│   │   │   │   │   └── app/
│   │   │   │   ├── terraform/ # Specific terraform modules
│   │   │   │   ├── BUILD
│   │   │   │   └── pyproject.toml
│   │   │   └── cli
│   │   │       ├── org-namespace
│   │   │       │   └── cli/
│   │   │       ├── terraform/ 
│   │   │       ├── BUILD
│   │   │       └── pyproject.toml
│   │   ├── BUILD
│   │   ├── Dockerfile
│   │   ├── Makefile
│   │   ├── poetry.lock
│   │   ├── poetry.toml
│   │   └── pyproject.toml
│   ├── release-pytorch/ # Similar structure to release-core/ (can use libs from release-core/)
│       └── [...]
├── .gitignore
├── pants.toml
└── README.md

The file

py/projects/resolve-core/pyproject.toml

uses path dependencies references to the relevant projects, and each project describes it's own set of dependencies. This way they can be worked on either in isolation using

poetry

or using

pants

goals (the CI ensures

pants generate-lockfiles ::

doesn't break)

[tool.poetry]
name = "resolve-core"
version = "0.1.0"

[tool.poetry.dependencies]
python = "^3.10"

[tool.poetry.group.app.dependencies]
namespace-app = { path = "projects/app", develop = true}

[tool.poetry.group.cli.dependencies]
namespace-cli = { path = "projects/cli", develop = true}

[tool.poetry.group.lib.dependencies]
namespace-lib = { path = "libs/lib1", develop = true}

I've tried to implement this (need to do some refactoring then I can share a working repo) and wanted to share a few insights (also had a few questions): •

pants tailor ::

I didn't have to modify any of the generated

BUILD

files to make this work (yet). • In the project I used as a starting point, the

python

version requirements was declared under the group

[tool.poetry.group.main.dependencies]

, which

poetry

seems to support.

pants

didn't though and understood

python

as a third-party dependency e.g.

# In py/release-core/pyproject.toml

# This fails when running `pants generate-lockfiles ::`
[tool.poetry.group.main.dependencies]
python = "^3.10"

# This works
[tool.poetry.dependencies]
python = "^3.10"

•

pants dependecies /path/to/file

fails to detect the dependency when trying to import a relative module using the absolute syntax, have any idea why ? e.g:

# app structure
# .
# ├── org_namespace
# │   └── app
# │       ├── __init__.py
# │       ├── BUILD
# │       ├── main.py
# │       └── test_main.py
# ├── BUILD
# └── pyproject.toml

# In py/release-core/projects/app/namespace/app/test_main.py
from .main import main # Successfully infers the dependency on main.py
from org_namespace.app.main import main # Code works but fails to infer the dependency

• If

resolve-pytorch

can use libs from

resolve-core

, would it make sense to put those libs elsewhere ? • This approach allows us to easily add a dependency using

poetry add

and leverage it's dependency resolution process. We'll still need to

pants generate-lockfiles ::

to integrate it into our resolve. But the cool thing is that we can work either with a poetry virtualenv or directly with Pants goals. • A

sandbox

project can add a relative path dependency to a lib and get going with poetry, which is nice. I'm looking for feedback on potential issues with this build setup, any insight or suggestion would be highly appreciated. Thank you for taking the time to review this! pantsbuild/pants