https://pantsbuild.org/ logo
Join Slack
Powered by
Hi all, I am evaluating the use of pants for a mon...
# general
t
Hi all, I am evaluating the use of pants for a mono repo that contains all of our internal libraries. The idea is to move from many single python poetry libraries to one mono repo that publishes the same libraries. Some of the libraries depend on libraries that are in the same project. As example, Got a logging library that every library depend. Then a stream library that some depend. No circle dependencies, just level dependencies that makes it really annoying/hard to keep the dependencies up to date. Is there any example out there of this use case? Thanks in advance!
h
Hi, welcome! This is a very typical use case. The Pants repo itself is structured this way. An idiomatic way to do this effectively is to not publish all these intermediate libraries, but consume their code directly from the repo. If you need to publish them for other reasons (e.g., consumption by other code in other repos) that's another matter. But one of the big advantages of a monorepo is not having to do that.
Just build your deployable artifacts, whatever they are, at HEAD
Does that make sense?
t
Hi Benjy, thanks the quick reply. I will need to publish them in separate packages, so the many consumers don’t require to get what they don’t need. Is that possible with Pants? Thanks again
so, as example. • library 
common
needs 
logger
and 
streamer
• library 
streamer
needs library 
logger
• library 
test-common
needs library 
logger
• library 
logger
needs 
pythonjsonlogger
Then the services might need some of those libraries or not. Example: • Service 
runner
needs 
logger
, 
streamer
, 
test-common
(on tests) and 
common
• Service 
collector
needs 
logger
and 
common
• etc Ideally I can put all libraries into the same repo, but still deploy them separated for the consumers to get them. Thanks in advance!
h
Are the consumers also in the same repo?
If so then you don't need to publish 
logger
for 
runner
to use it. They are in the same repo!
t
no, different repos. In the future they “might” be but can’t migrate that many repos into one in one go. (talking about 100s of repos publishing serverless stacks)
I guess I can package them as submodules and just deploy one version… I will look into that and, well, probably ask a few questions when I get stuck with other stuff. Will try to document the process so hopefully helps others… if that submodules option works. The main driver is size that, as a side effect, affects the startup time of lambdas etc.
h
gotcha
so you can still do both - publish wheels for outside consumers, but consume internally at HEAD
Do you have existing 
setup.py
or 
pyproject.toml
for all these wheels you're publishing?
t
pyproject.toml
h
Basically something like 
./pants package ::
will build all of them, 
./pants --changed-since=<SHA> --changed-dependees=transitive package
will build all of the ones that are affected by changes since <SHA>
And so on
But it sounds like part of what you're asking about is automatically managing the dependency versions? So that when you build a new 
logging
you don't have to manually update 
common/pyproject.toml
with the new version of 
logging
?
👍 1
t
gotcha, I guess I will need to have then a pyproject.toml per project (what makes sense). I will play around with it, and probably get back once I get stuck with some nonsense 😄 Thanks a lot!
and yes to those 2 last questions. I assume we will need to put all the 3rd party dependencies in a different toml file so they can be shared, but will play around as I said. Thanks again for your help!
h
I assumed you already had a pyproject.toml per project?
You can also let Pants generate setup.py for you
👍 1
Then it will automatically use the right versions of dependencies
t
yeah, got many little libraries that are doing my (and a few people) head, so planning to move them to a mono repo… and then see if we can do the same for our services. The plan is to have a huge mono repo so every developer can contribute to every project and avoid the “team silo” that mini repos create.
for info, found an interesting issue in the python example with pythonjsonlogger module mappings. https://github.com/pantsbuild/example-python/issues/120 Let me know if you need further info to clarify and/or solve it. happy to help, and also happy to dig a bit on the code if you give me some pointers 😉
11 Views
Open in Slack
PreviousNext
Powered by