I'm trying to organize a monorepo containing a bunch of Python packages. Each package is managed by Poetry and thus has its own pyproject.toml and poetry.lock. An issue I'm running into is that when a new dependency is added to package A, it is necessary to run

poetry install

for all packages that specify package A as a path dependency so that their poetry.lock is updated. I learned about Pants in a comment on an issue about monorepos on the Poetry issue tracker. I was hoping Pants would be able to solve the problem above. Does it? It's not clear to me what "Poetry support" in Pants encompasses. From, early experiments with Pants 2.6 it doesn't seem that Pants considers the Poetry lock files. I'm hoping someone can shed some light on this here. Thanks! 🙏

I think the poetry docs say that all the packages should have a lock. This is inline with advice from javascript's npm. However, in Rust when you specify dependencies in Cargo you only provide a lock for applications and not for libraries so applications can try to tease out some compatible versions that might not be possible using pinned versions of transient dependencies.

Not sure I get your point, but it's indeed not really necessary for dependencies to have a poetry.lock file. To improve my question:

poetry install

needs to run for the top-level package when a pyproject.toml for a dependency changes.

Ah yeah, that's the command. Sorry

You can test some stuff out here: https://github.com/ehiggs/pants-poetry-test-repos

I did stumble upon that repo (thanks!). Does that setup somehow allow detecting changes to dependencies and updating the top-level package? If so, how?

no, not yet. it's very basic at the moment. I've been trying to work on a few other scenarios when i get time: 1. all poetry and no pants 2. all poetry, but one pants package to simulate a migration in progress 3. docker-compose, where is the best place for Dockerfile and managing deps (all in one? each application getting its own Dockerfile?) I think your scenario is also v. important.

I'm getting the impression it makes more sense to replace Poetry with Pants for the monorepo use case? Would that then handle creating individual distribution packages?

I dont have an answer to that as I'm also relatively new to pants

I am quite new to Pants as well; we’ve just finished integrating and now use Pants in a Python monorepo and have multiple artifacts that are produced (wheels and PEX files). We don’t use Poetry at all (there are some standalone repos that do, but we see no need for it in the monorepo).

Would that then handle creating individual distribution packages?

Sorry, what do you mean by a “distribution package”?

the extent of the poetry support is that the relevant poetry sections of

pyproject.toml

files are consumed and used as the input to resolves. there are docs about it, but only on the 2.6 version of the docsite (not stable yet!): https://www.pantsbuild.org/v2.6/docs/python-third-party-dependencies#poetry-integration

You are looking for something that is currently not covered by pants or poetry. Pants just gained the ability to pull poetry styled requirements from

pyproject.toml

, but it cannot read the poetry lock file at this point. There is no tooling to help with a parent

pyproject.toml

and children

pyproject.toml

files. You might need to write a poetry plugin to handle propogating dependency updates through layered

pyproject.toml

files in a monorepo like that. Or, you could have one super

pyproject.toml

file and lock file where you manage the superset of dependencies for all your libs. Then, you can use pants to see which dependencies are used by which packages. You could use pants to generate requirements.txt in each of those packages, and you can use pants to build a python distribution (wheel) which includes generating a setup.py file with all the relevant requirements embedded in it.

yea. while Pants can consume multiple

pyproject.toml

files, if they each declare the same dependencies, you’ll end up needing to explicitly choose the version you want in your BUILD files

You might be interested in how I set it up. I didn't solve everything the pants way but it solves all my itches with regards to releasing multiple inter-dependant packages from a monorepo: https://github.com/Incognito/python-architecture-linter/blob/2a6a0be9a4d1558eb3006ecc65511944f0c82dd7/.github/workflows/main.yml#L63 Not perfect, has some rough edges but everything is working. I use pants for the package release and poetry for everything else.

nice! fwiw, with a small amount of Pants configuration, your “lint typecheck test” steps could become

./pants lint typecheck test ::

, and the linters and tests would run concurrently… and run the same way in CI as locally.

Thanks for the hint. I'm new to pants but eventually I intend to try and migrate everything over to it.

Thanks all for your feedback! Initially, I found Pants very confusing, but perhaps that's because I was naively expecting it to solve the issues I had with this specific multi-`pyproject.toml`/`poetry.lock` monorepo. Things are becoming more clear now. I'll do some more tests with Pants and see where that takes me...

good idea: done.