https://pantsbuild.org/ logo
Join Slack
Powered by
Anyone experience for poetry and pants?
# general
c
Anyone experience for poetry and pants?
h
Welcome! @steep-breakfast-98857 is actually working on Poetry support in Pants right now. I think it's almost done. What kind of functionality were you looking to have?
c
Currently we have the issue that we have several django apps that are deployed on our private pypi and distributed to pypi. Since there share a lot of characteristics, a mono repo seems to be the obvious choice - currently we have have one repo for each app. We use Poetry for dependency resolution and it seems "natural" to me. I am still looking into a starting point, as I have not found anybody who actually build a django monorepo with pants, but I think it is the right way to go for us.
h
My company has a monorepo with many Django apps which share underlying infrastructure, just as you describe.
And we use Pants for that.
But with pip, not poetry.
However poetry support is coming, as I mentioned
One interesting thing to note is that you don't need a separate set of requirements for each of your Django apps. You can have one global one for your repo, and Pants will package just the ones that each app actually requires.
We have been working on an Django example repo
It's just been on the back burner for a while
I should really get that going again
I'm actually giving a workshop on exactly this topic at DjangoCon EU in June
Anyway, we're happy to help you get Pants set up for this! We have a lot of Django-friendly features (such as dependency inference from strings, because module name references are common in 
settings.py
.)
c
Thanks for your answer.
I just started yesterday with this project without any django and i see how to get started. It is quite hard for me to say where I am struggling, but currently I face two issues: • Where to put the django project and how to use it (the apps act like libraries which i can use and distribute to pypi, i get that) but I cannot seem to get started on the project, e.g. where to put the settings, where to put the manage.py, do i create a pex_executable out of the project? • How do I integrate the dependencies in my IDE? Currently I have a virtualenvironment, where i can simply load the correct python version and the IDE takes care of it. How do I do this with pants?
h
How do I integrate the dependencies in my IDE?
You can use something like this https://www.pantsbuild.org/docs/python-third-party-dependencies#tip-set-up-a-virtual-environment-optional to generate a virtualenv, then load it same as you're doing now. Or continue to use the venv from Poetry, which works unless you start adding 
python_requirement_library
targets inlined in BUILD files (usually no need to) Note that this 3rd party dependency handling is going to be improved very very soon. I have a design doc out to redesign several things (https://docs.google.com/document/d/1bCYb0UQZx9a-9tAagydCN_z3826QRvz_3aVnXKSNTJw/edit if you're curious)
where to put the settings, where to put the manage.py, do i create a pex_executable out of the project?
I wish we had an example repo already and that I understood Django better. But looking at Toolchain's code, we have several distinct Django projects, and each gets its own 
manage.py
and 
settings.py
, e.g. 
projectA/manage.py
and 
projectA/settings.py
. Then yeah, a 
pex_binary
for the 
mangage.py
like this:
Copy code
pex_binary(
  name="manage",
  entry_point="manage.py",
  execution_mode="unzip",
)
I can try to dig more what happens from there, and @polite-garden-50641 has more experience. (Benjy too, but sounds like something came up that he might not get to this right away)
c
Thank you for the quick response. Really helpful. I will take a look at the links and get back to you. Already a big thanks to @hundreds-father-404 and @happy-kitchen-89482. Really appreciate the help.
❤️ 1
One question that I already have: If every library has its own set of dependencies, and we are packaging them to our private Pypi, then I need something like 
python_requirement_library
right?
h
Not necessarily
Where you might want to end up is having a single 
requirements.txt
for the entire monorepo (with a BUILD file containing a 
python_requirements()
macro call that wraps that 
requirements.txt
file)
And these are just your repo's external dependencies, i.e., code from other repos
Then you have 
python_library()
targets for each package (
./pants tailor
will generate these for you).
h
You should not need to:
If every library has its own set of dependencies
Pants calculates this for you via dependency inference, picked from the universe of requirements knows about from things like your requirements.txt (or soon, from reading pyproject.toml). Run 
./pants dependencies --transitive path/to/file.py
for example
we are packaging them to our private Pypi,
You can set a custom repository: https://www.pantsbuild.org/docs/python-third-party-dependencies#using-custom-repositories
h
And for each of your Django apps you probably want a 
python_distribution()
target which tells Pants how to generate a dist for it that you can publish to your internal PyPI, or to real PyPI.
Note that when you run 
./pants package
on that 
python_distribution
Pants will knit together the correct dependencies for you, and inject them into the 
setup()
call.
So you don't maintain setup.py files at all
One thing Pants doesn't do yet is call Twine to upload those packaged dists. It writes them out under 
dist/
and you have to publish them outside of Pants.
Sorry, this is a big brain dump, hope it's not too confusing.
c
Okay I think I get it. So the dependencies for the library I put into the python_distribution.dependencies for distributing them to pypi and for local development I keep on virtual environment with all depenedencies?
Confusing yes, but I am starting to get my head around it 🙂
h
Note that as Eric mentioned above, we distribute our Django apps as 
.pex
files, via 
pex_binary
. But it sounds like you want them to be wheels, so you'd use 
python_distribution
.
c
yes i understand that. I need both, one for local development, one for distributing to clients. (does that make sense?)
h
Nope, it's simpler than that
I'm just not doing the best job of explaining it
Let me try again 🙂
c
Thanks for your patience with me 🙂
h
Thanks for bearing with us!
So - 
python_requirements()
represents the 3rdparty deps. Typically you only need one of those in the entire repo, to wrap your 
requirements.txt
. You do not put references to the published versions of in-repo code in that requirements.txt. All dependencies between code in the repo are expressed as internal deps.
What other target types do we have?
python_library()
represents internal code, usually one per directory
python_tests()
represents tests, also usually one per directory
./pants tailor
will generate these for you, mostly correctly...
If you didn't have to package and distribute your code, this would be all you would need,
But of course you do need to package and distribute your code.
There are a few ways to do this - pex files, aws lambda, wheels etc.
It sounds like your chosen method is wheels?
So, how to we get Pants to produce wheels?
we use a 
python_distribution()
target
When you run 
./pants package
on that target, under the covers Pants runs 
setuptools.setup()
on it, like you're probably used to.
But there is no setup.py to provide arguments to setup!
Pants generates those arguments from a combination of "things it can deduce from the repo" and "things you explicitly provide in the 
python_distribution()
target.
But one of the things it can generate is dependencies.
It maps every 
python_library
to the wheel that publishes that code
so if code in wheel B depends on code in wheel A, it will add a dependency from B to A when it generates the setup arguments
Note that how python libraries map to wheels is a little nuanced, there are details here: https://www.pantsbuild.org/docs/python-distributions
c
Thanks for the elaborate explanation - I think I understand all of that, at least it makes sense. The only small thing I do not understand is, how does python_distribution know that this library is dependand on an external dependency, if I have a constraints.txt at the root folder for the entire project.
h
Most of the time Pants infers those dependencies by looking at import statements
if you have a dep that can't be inferred you can add it manually
c
really?
h
(This is what I meant by saying run 
./pants dependencies path/to/file.py
and 
./pants dependencies --transitive path/to/file.py
)
c
so if i say: from django.db import models pants knows that we need django?
h
correct
the 
python_requirements()
macro essentially takes your 
requirements.txt
and turns each req into a target, so if it sees 
Django>=3.0.0
it turns that into a target called 
django
in the same directory.
c
alright, and only if pants does not deduce this or if i need to pin it to a particular version, then i add it in the dependencies section of python_distribution?
➕ 1
h
of a 
python_library
python_distribution
is just "how do I publish code", dependency information for some code lives on the 
python_library
for that code.
c
ah ok - so pants doesn't do this automatically. I checks my requirements.txt and then deduces which it needs here?
Ok then i think i understand it.
h
Pants does it automatically most of the time
Sorry, I mean yes you need to have 
Django
in your requirements.txt for Pants to be able to infer a dep on it if it sees 
from django import...
somewhere.
c
Yeah i got that
One last question
h
But you don't need to add an explicit dep in the 
python_library
of the importing code
You only need to do that if it's a dep that can't be inferred by looking at imports.
c
You mentioned that each app should have its own manage.py and own settings.py, etc. why is that?
h
And those do happen with Django, because of runtime loading by name (e.g., module strings in settings.py)
Internally we have a ton of shared code, and then each Django project's manage.py and settings.py are very lightweight and just import from shared underlying files, tweaking what they need.
But "each Django project needs manage.py and settings.py" is a Django necessity, not a Pants one
That's just how Django works
c
yes of course each django project needs a manage.py
h
There's a bunch of per-service boilerplate that Django expects
c
But what i wanted to achieve is, that i have my 20 apps, and then one project for developing them, e.g. running them.
h
So this is "apps" and "projects" in the Django sense?
c
because currently we have 20 apps / 20 repositories / 20 test_projects
yes
h
In Django "project" more or less means "service"
Ah, so I misunderstood what you currently have
in that case yeah, you only need one manage+settings
One per different type of service
In fact, do you need to publish your individual apps at all?
c
ok. that makes sense then.
h
This might be much easier than I thought
I mistakenly thought you had 20 distinct services
c
yes i do. because we have different clients for our ERP software and each of them use a different combination of our django apps / libaries
h
I see
c
so on a client side we have one repository for each client which just installs the apps and have their customization. but i am completely fine with seperating them, at least for now away from the mono repo
h
So one thing to be aware of, and https://www.pantsbuild.org/docs/python-distributions explains this, is that you don't want the same shared library code published in multiple wheels
so it needs to be in its own wheel
that the per-app wheels depend on
I should go to sleep now, but let us know how it goes, happy to answer more questions
those doc pages have a lot of useful stuff
c
I think I understand now how to continue. Thanks a lot for your help. Good night 🙂
❤️ 1
5 Views
Open in Slack
PreviousNext
Powered by