https://pantsbuild.org/ logo
#general
Title
# general
q

quaint-gold-40000

07/22/2021, 9:31 AM
I'm trying to organize a monorepo containing a bunch of Python packages. Each package is managed by Poetry and thus has its own pyproject.toml and poetry.lock. An issue I'm running into is that when a new dependency is added to package A, it is necessary to run
poetry install
for all packages that specify package A as a path dependency so that their poetry.lock is updated. I learned about Pants in a comment on an issue about monorepos on the Poetry issue tracker. I was hoping Pants would be able to solve the problem above. Does it? It's not clear to me what "Poetry support" in Pants encompasses. From, early experiments with Pants 2.6 it doesn't seem that Pants considers the Poetry lock files. I'm hoping someone can shed some light on this here. Thanks! 🙏
p

powerful-boots-1234

07/22/2021, 9:33 AM
I think the poetry docs say that all the packages should have a lock. This is inline with advice from javascript's npm. However, in Rust when you specify dependencies in Cargo you only provide a lock for applications and not for libraries so applications can try to tease out some compatible versions that might not be possible using pinned versions of transient dependencies.
Given that any python dep should be installable as a wheel; it would be prebuilt so a lockfile makes sense. but in a monorepo, does it make sense if the only consumers are in the same repo? 🤷
Anyway regarding your question(s):
An issue I'm running into is that when a new dependency is added to package A, it is necessary to run 
poetry install
 for all packages that specify package A as a path dependency so that their poetry.lock is updated.
This is correct. not that you can do
poetry install library-name==version
and it only touches that library and it's transient dependencies. It's annoying but I'm not sure whether poetry or pants considers this a bug.
I was hoping Pants would be able to solve the problem above. Does it? It's not clear to me what "Poetry support" in Pants encompasses. From, early experiments with Pants 2.6 it doesn't seem that Pants considers the Poetry lock files. I'm hoping someone can shed some light on this here. Thanks!
You can test some stuff out here: https://github.com/ehiggs/pants-poetry-test-repos As 2.6.0 is still in rc stage there are plenty of workflows that need maturing and this only happens when people push on it.
q

quaint-gold-40000

07/22/2021, 9:40 AM
Not sure I get your point, but it's indeed not really necessary for dependencies to have a poetry.lock file. To improve my question:
poetry install
needs to run for the top-level package when a pyproject.toml for a dependency changes.
This is correct. not that you can do 
poetry install library-name==version
  and it only touches that library and it's transient dependencies. It's annoying but I'm not sure whether poetry or pants considers this a bug.
You can do
poetry update <depencency>
though to limit changes to the venv and lock file.
p

powerful-boots-1234

07/22/2021, 9:42 AM
Ah yeah, that's the command. Sorry
q

quaint-gold-40000

07/22/2021, 9:43 AM
You can test some stuff out here: https://github.com/ehiggs/pants-poetry-test-repos
I did stumble upon that repo (thanks!). Does that setup somehow allow detecting changes to dependencies and updating the top-level package? If so, how?
Before learning about Pants I considered writing a script to scan for pyproject.toml files, discover dependencies and check whether an update is needed. But that seems like asking for trouble. Alternatively, I think a lot of complexity can be avoided by moving all packages into a single large package, since the individual packages are unlikely to ever be separately distributed.
p

powerful-boots-1234

07/22/2021, 9:48 AM
no, not yet. it's very basic at the moment. I've been trying to work on a few other scenarios when i get time: 1. all poetry and no pants 2. all poetry, but one pants package to simulate a migration in progress 3. docker-compose, where is the best place for Dockerfile and managing deps (all in one? each application getting its own Dockerfile?) I think your scenario is also v. important.
q

quaint-gold-40000

07/22/2021, 12:25 PM
I'm getting the impression it makes more sense to replace Poetry with Pants for the monorepo use case? Would that then handle creating individual distribution packages?
p

powerful-boots-1234

07/22/2021, 2:26 PM
I dont have an answer to that as I'm also relatively new to pants
f

fresh-cat-90827

07/22/2021, 2:57 PM
I am quite new to Pants as well; we’ve just finished integrating and now use Pants in a Python monorepo and have multiple artifacts that are produced (wheels and PEX files). We don’t use Poetry at all (there are some standalone repos that do, but we see no need for it in the monorepo).
Would that then handle creating individual distribution packages?
Sorry, what do you mean by a “distribution package”?
w

witty-crayon-22786

07/22/2021, 3:57 PM
the extent of the poetry support is that the relevant poetry sections of
pyproject.toml
files are consumed and used as the input to resolves. there are docs about it, but only on the 2.6 version of the docsite (not stable yet!): https://www.pantsbuild.org/v2.6/docs/python-third-party-dependencies#poetry-integration
p

proud-dentist-22844

07/22/2021, 3:58 PM
You are looking for something that is currently not covered by pants or poetry. Pants just gained the ability to pull poetry styled requirements from
pyproject.toml
, but it cannot read the poetry lock file at this point. There is no tooling to help with a parent
pyproject.toml
and children
pyproject.toml
files. You might need to write a poetry plugin to handle propogating dependency updates through layered
pyproject.toml
files in a monorepo like that. Or, you could have one super
pyproject.toml
file and lock file where you manage the superset of dependencies for all your libs. Then, you can use pants to see which dependencies are used by which packages. You could use pants to generate requirements.txt in each of those packages, and you can use pants to build a python distribution (wheel) which includes generating a setup.py file with all the relevant requirements embedded in it.
w

witty-crayon-22786

07/22/2021, 3:59 PM
yea. while Pants can consume multiple
pyproject.toml
files, if they each declare the same dependencies, you’ll end up needing to explicitly choose the version you want in your BUILD files
feedback on the docs linked above very welcome!
and yes: support for multiple lockfiles is coming soon: currently only one is supported, which doesn’t allow for conflicts between your
pyproject.toml
or other requirements.
@quaint-gold-40000: with regard to the “poetry install” command: the equivalent in Pants depends on whether you are using a lockfile (which is optional right now): • if you are not using a lockfile, there are no manual steps after changing your requirements/pyproject.toml: just run pants as you usually would, and it will re-resolve. • if you are using a lockfile, you’ll need to regenerate it using the same instruction you use to create it, and then run pants
the lockfile/resolve overhaul that is going on now will eventually remove that manual step.
n

nice-france-74763

07/22/2021, 6:19 PM
You might be interested in how I set it up. I didn't solve everything the pants way but it solves all my itches with regards to releasing multiple inter-dependant packages from a monorepo: https://github.com/Incognito/python-architecture-linter/blob/2a6a0be9a4d1558eb3006ecc65511944f0c82dd7/.github/workflows/main.yml#L63 Not perfect, has some rough edges but everything is working. I use pants for the package release and poetry for everything else.
❤️ 1
w

witty-crayon-22786

07/22/2021, 6:22 PM
nice! fwiw, with a small amount of Pants configuration, your “lint typecheck test” steps could become
./pants lint typecheck test ::
, and the linters and tests would run concurrently… and run the same way in CI as locally.
(and in future all of lint/typecheck/test will run concurrently as well, rather than sequentially: that’s #10542)
n

nice-france-74763

07/22/2021, 6:32 PM
Thanks for the hint. I'm new to pants but eventually I intend to try and migrate everything over to it.
đź‘Ť 1
q

quaint-gold-40000

07/26/2021, 10:02 AM
Thanks all for your feedback! Initially, I found Pants very confusing, but perhaps that's because I was naively expecting it to solve the issues I had with this specific multi-`pyproject.toml`/`poetry.lock` monorepo. Things are becoming more clear now. I'll do some more tests with Pants and see where that takes me...
@witty-crayon-22786 I did find the docs on Poetry integration early in my investigation. The only feedback I can give on them for now it that it might help explicitly mentioning that
poetry.lock
files are ignored by Pants.
đź‘Ť 1
w

witty-crayon-22786

07/26/2021, 3:56 PM
good idea: done.
🙏 1
5 Views