https://pantsbuild.org/ logo
#general
Title
# general
t

thousands-france-27863

01/16/2023, 11:20 AM
Hi all, I am evaluating the use of pants for a mono repo that contains all of our internal libraries. The idea is to move from many single python poetry libraries to one mono repo that publishes the same libraries. Some of the libraries depend on libraries that are in the same project. As example, Got a logging library that every library depend. Then a stream library that some depend. No circle dependencies, just level dependencies that makes it really annoying/hard to keep the dependencies up to date. Is there any example out there of this use case? Thanks in advance!
h

happy-kitchen-89482

01/16/2023, 12:10 PM
Hi, welcome! This is a very typical use case. The Pants repo itself is structured this way. An idiomatic way to do this effectively is to not publish all these intermediate libraries, but consume their code directly from the repo. If you need to publish them for other reasons (e.g., consumption by other code in other repos) that's another matter. But one of the big advantages of a monorepo is not having to do that.
Just build your deployable artifacts, whatever they are, at HEAD
Does that make sense?
t

thousands-france-27863

01/16/2023, 12:13 PM
Hi Benjy, thanks the quick reply. I will need to publish them in separate packages, so the many consumers don’t require to get what they don’t need. Is that possible with Pants? Thanks again
so, as example. • library
common
needs
logger
and
streamer
• library
streamer
needs library
logger
• library
test-common
needs library
logger
• library
logger
needs
pythonjsonlogger
Then the services might need some of those libraries or not. Example: • Service
runner
needs
logger
,
streamer
,
test-common
(on tests) and
common
• Service
collector
needs
logger
and
common
• etc Ideally I can put all libraries into the same repo, but still deploy them separated for the consumers to get them. Thanks in advance!
h

happy-kitchen-89482

01/16/2023, 1:22 PM
Are the consumers also in the same repo?
If so then you don't need to publish
logger
for
runner
to use it. They are in the same repo!
t

thousands-france-27863

01/16/2023, 2:17 PM
no, different repos. In the future they “might” be but can’t migrate that many repos into one in one go. (talking about 100s of repos publishing serverless stacks)
I guess I can package them as submodules and just deploy one version… I will look into that and, well, probably ask a few questions when I get stuck with other stuff. Will try to document the process so hopefully helps others… if that submodules option works. The main driver is size that, as a side effect, affects the startup time of lambdas etc.
h

happy-kitchen-89482

01/16/2023, 4:12 PM
gotcha
so you can still do both - publish wheels for outside consumers, but consume internally at HEAD
Do you have existing
setup.py
or
pyproject.toml
for all these wheels you're publishing?
t

thousands-france-27863

01/16/2023, 4:13 PM
pyproject.toml
h

happy-kitchen-89482

01/16/2023, 4:14 PM
Basically something like
./pants package ::
will build all of them,
./pants --changed-since=<SHA> --changed-dependees=transitive package
will build all of the ones that are affected by changes since <SHA>
And so on
But it sounds like part of what you're asking about is automatically managing the dependency versions? So that when you build a new
logging
you don't have to manually update
common/pyproject.toml
with the new version of
logging
?
👍 1
t

thousands-france-27863

01/16/2023, 4:33 PM
gotcha, I guess I will need to have then a pyproject.toml per project (what makes sense). I will play around with it, and probably get back once I get stuck with some nonsense 😄 Thanks a lot!
and yes to those 2 last questions. I assume we will need to put all the 3rd party dependencies in a different toml file so they can be shared, but will play around as I said. Thanks again for your help!
h

happy-kitchen-89482

01/16/2023, 4:41 PM
I assumed you already had a pyproject.toml per project?
You can also let Pants generate setup.py for you
👍 1
Then it will automatically use the right versions of dependencies
t

thousands-france-27863

01/16/2023, 4:44 PM
yeah, got many little libraries that are doing my (and a few people) head, so planning to move them to a mono repo… and then see if we can do the same for our services. The plan is to have a huge mono repo so every developer can contribute to every project and avoid the “team silo” that mini repos create.
for info, found an interesting issue in the python example with pythonjsonlogger module mappings. https://github.com/pantsbuild/example-python/issues/120 Let me know if you need further info to clarify and/or solve it. happy to help, and also happy to dig a bit on the code if you give me some pointers 😉
11 Views