https://pantsbuild.org/ logo
#general
Title
# general
c

creamy-truck-92790

05/06/2021, 11:40 AM
Anyone experience for poetry and pants?
h

happy-kitchen-89482

05/06/2021, 2:33 PM
Welcome! @steep-breakfast-98857 is actually working on Poetry support in Pants right now. I think it's almost done. What kind of functionality were you looking to have?
c

creamy-truck-92790

05/06/2021, 2:51 PM
Currently we have the issue that we have several django apps that are deployed on our private pypi and distributed to pypi. Since there share a lot of characteristics, a mono repo seems to be the obvious choice - currently we have have one repo for each app. We use Poetry for dependency resolution and it seems "natural" to me. I am still looking into a starting point, as I have not found anybody who actually build a django monorepo with pants, but I think it is the right way to go for us.
h

happy-kitchen-89482

05/06/2021, 4:18 PM
My company has a monorepo with many Django apps which share underlying infrastructure, just as you describe.
And we use Pants for that.
But with pip, not poetry.
However poetry support is coming, as I mentioned
One interesting thing to note is that you don't need a separate set of requirements for each of your Django apps. You can have one global one for your repo, and Pants will package just the ones that each app actually requires.
We have been working on an Django example repo
It's just been on the back burner for a while
I should really get that going again
I'm actually giving a workshop on exactly this topic at DjangoCon EU in June
Anyway, we're happy to help you get Pants set up for this! We have a lot of Django-friendly features (such as dependency inference from strings, because module name references are common in
settings.py
.)
c

creamy-truck-92790

05/07/2021, 6:16 AM
Thanks for your answer.
I just started yesterday with this project without any django and i see how to get started. It is quite hard for me to say where I am struggling, but currently I face two issues: ā€¢ Where to put the django project and how to use it (the apps act like libraries which i can use and distribute to pypi, i get that) but I cannot seem to get started on the project, e.g. where to put the settings, where to put the manage.py, do i create a pex_executable out of the project? ā€¢ How do I integrate the dependencies in my IDE? Currently I have a virtualenvironment, where i can simply load the correct python version and the IDE takes care of it. How do I do this with pants?
h

hundreds-father-404

05/07/2021, 6:28 AM
How do I integrate the dependencies in my IDE?
You can use something like this https://www.pantsbuild.org/docs/python-third-party-dependencies#tip-set-up-a-virtual-environment-optional to generate a virtualenv, then load it same as you're doing now. Or continue to use the venv from Poetry, which works unless you start adding
python_requirement_library
targets inlined in BUILD files (usually no need to) Note that this 3rd party dependency handling is going to be improved very very soon. I have a design doc out to redesign several things (https://docs.google.com/document/d/1bCYb0UQZx9a-9tAagydCN_z3826QRvz_3aVnXKSNTJw/edit if you're curious)
where to put the settings, where to put the manage.py, do i create a pex_executable out of the project?
I wish we had an example repo already and that I understood Django better. But looking at Toolchain's code, we have several distinct Django projects, and each gets its own
manage.py
and
settings.py
, e.g.
projectA/manage.py
and
projectA/settings.py
. Then yeah, a
pex_binary
for the
mangage.py
like this:
Copy code
pex_binary(
  name="manage",
  entry_point="manage.py",
  execution_mode="unzip",
)
I can try to dig more what happens from there, and @polite-garden-50641 has more experience. (Benjy too, but sounds like something came up that he might not get to this right away)
c

creamy-truck-92790

05/07/2021, 6:34 AM
Thank you for the quick response. Really helpful. I will take a look at the links and get back to you. Already a big thanks to @hundreds-father-404 and @happy-kitchen-89482. Really appreciate the help.
ā¤ļø 1
One question that I already have: If every library has its own set of dependencies, and we are packaging them to our private Pypi, then I need something like
python_requirement_library
right?
h

happy-kitchen-89482

05/07/2021, 6:36 AM
Not necessarily
Where you might want to end up is having a single
requirements.txt
for the entire monorepo (with a BUILD file containing a
python_requirements()
macro call that wraps that
requirements.txt
file)
And these are just your repo's external dependencies, i.e., code from other repos
Then you have
python_library()
targets for each package (
./pants tailor
will generate these for you).
h

hundreds-father-404

05/07/2021, 6:38 AM
You should not need to:
If every library has its own set of dependencies
Pants calculates this for you via dependency inference, picked from the universe of requirements knows about from things like your requirements.txt (or soon, from reading pyproject.toml). Run
./pants dependencies --transitive path/to/file.py
for example
we are packaging them to our private Pypi,
You can set a custom repository: https://www.pantsbuild.org/docs/python-third-party-dependencies#using-custom-repositories
h

happy-kitchen-89482

05/07/2021, 6:39 AM
And for each of your Django apps you probably want a
python_distribution()
target which tells Pants how to generate a dist for it that you can publish to your internal PyPI, or to real PyPI.
Note that when you run
./pants package
on that
python_distribution
Pants will knit together the correct dependencies for you, and inject them into the
setup()
call.
So you don't maintain setup.py files at all
One thing Pants doesn't do yet is call Twine to upload those packaged dists. It writes them out under
dist/
and you have to publish them outside of Pants.
Sorry, this is a big brain dump, hope it's not too confusing.
c

creamy-truck-92790

05/07/2021, 6:42 AM
Okay I think I get it. So the dependencies for the library I put into the python_distribution.dependencies for distributing them to pypi and for local development I keep on virtual environment with all depenedencies?
Confusing yes, but I am starting to get my head around it šŸ™‚
h

happy-kitchen-89482

05/07/2021, 6:42 AM
Note that as Eric mentioned above, we distribute our Django apps as
.pex
files, via
pex_binary
. But it sounds like you want them to be wheels, so you'd use
python_distribution
.
c

creamy-truck-92790

05/07/2021, 6:43 AM
yes i understand that. I need both, one for local development, one for distributing to clients. (does that make sense?)
h

happy-kitchen-89482

05/07/2021, 6:44 AM
Nope, it's simpler than that
I'm just not doing the best job of explaining it
Let me try again šŸ™‚
c

creamy-truck-92790

05/07/2021, 6:45 AM
Thanks for your patience with me šŸ™‚
h

happy-kitchen-89482

05/07/2021, 6:45 AM
Thanks for bearing with us!
So -
python_requirements()
represents the 3rdparty deps. Typically you only need one of those in the entire repo, to wrap your
requirements.txt
. You do not put references to the published versions of in-repo code in that requirements.txt. All dependencies between code in the repo are expressed as internal deps.
What other target types do we have?
python_library()
represents internal code, usually one per directory
python_tests()
represents tests, also usually one per directory
./pants tailor
will generate these for you, mostly correctly...
If you didn't have to package and distribute your code, this would be all you would need,
But of course you do need to package and distribute your code.
There are a few ways to do this - pex files, aws lambda, wheels etc.
It sounds like your chosen method is wheels?
So, how to we get Pants to produce wheels?
we use a
python_distribution()
target
When you run
./pants package
on that target, under the covers Pants runs
setuptools.setup()
on it, like you're probably used to.
But there is no setup.py to provide arguments to setup!
Pants generates those arguments from a combination of "things it can deduce from the repo" and "things you explicitly provide in the
python_distribution()
target.
But one of the things it can generate is dependencies.
It maps every
python_library
to the wheel that publishes that code
so if code in wheel B depends on code in wheel A, it will add a dependency from B to A when it generates the setup arguments
Note that how python libraries map to wheels is a little nuanced, there are details here: https://www.pantsbuild.org/docs/python-distributions
c

creamy-truck-92790

05/07/2021, 6:52 AM
Thanks for the elaborate explanation - I think I understand all of that, at least it makes sense. The only small thing I do not understand is, how does python_distribution know that this library is dependand on an external dependency, if I have a constraints.txt at the root folder for the entire project.
h

happy-kitchen-89482

05/07/2021, 6:53 AM
Most of the time Pants infers those dependencies by looking at import statements
if you have a dep that can't be inferred you can add it manually
c

creamy-truck-92790

05/07/2021, 6:53 AM
really?
h

hundreds-father-404

05/07/2021, 6:54 AM
(This is what I meant by saying run
./pants dependencies path/to/file.py
and
./pants dependencies --transitive path/to/file.py
)
c

creamy-truck-92790

05/07/2021, 6:54 AM
so if i say: from django.db import models pants knows that we need django?
h

happy-kitchen-89482

05/07/2021, 6:54 AM
correct
the
python_requirements()
macro essentially takes your
requirements.txt
and turns each req into a target, so if it sees
Django>=3.0.0
it turns that into a target called
django
in the same directory.
c

creamy-truck-92790

05/07/2021, 6:55 AM
alright, and only if pants does not deduce this or if i need to pin it to a particular version, then i add it in the dependencies section of python_distribution?
āž• 1
h

happy-kitchen-89482

05/07/2021, 6:55 AM
of a
python_library
python_distribution
is just "how do I publish code", dependency information for some code lives on the
python_library
for that code.
c

creamy-truck-92790

05/07/2021, 6:56 AM
ah ok - so pants doesn't do this automatically. I checks my requirements.txt and then deduces which it needs here?
Ok then i think i understand it.
h

happy-kitchen-89482

05/07/2021, 6:56 AM
Pants does it automatically most of the time
Sorry, I mean yes you need to have
Django
in your requirements.txt for Pants to be able to infer a dep on it if it sees
from django import...
somewhere.
c

creamy-truck-92790

05/07/2021, 6:57 AM
Yeah i got that
One last question
h

happy-kitchen-89482

05/07/2021, 6:57 AM
But you don't need to add an explicit dep in the
python_library
of the importing code
You only need to do that if it's a dep that can't be inferred by looking at imports.
c

creamy-truck-92790

05/07/2021, 6:57 AM
You mentioned that each app should have its own manage.py and own settings.py, etc. why is that?
h

happy-kitchen-89482

05/07/2021, 6:58 AM
And those do happen with Django, because of runtime loading by name (e.g., module strings in settings.py)
Internally we have a ton of shared code, and then each Django project's manage.py and settings.py are very lightweight and just import from shared underlying files, tweaking what they need.
But "each Django project needs manage.py and settings.py" is a Django necessity, not a Pants one
That's just how Django works
c

creamy-truck-92790

05/07/2021, 6:59 AM
yes of course each django project needs a manage.py
h

happy-kitchen-89482

05/07/2021, 6:59 AM
There's a bunch of per-service boilerplate that Django expects
c

creamy-truck-92790

05/07/2021, 7:00 AM
But what i wanted to achieve is, that i have my 20 apps, and then one project for developing them, e.g. running them.
h

happy-kitchen-89482

05/07/2021, 7:00 AM
So this is "apps" and "projects" in the Django sense?
c

creamy-truck-92790

05/07/2021, 7:00 AM
because currently we have 20 apps / 20 repositories / 20 test_projects
yes
h

happy-kitchen-89482

05/07/2021, 7:00 AM
In Django "project" more or less means "service"
Ah, so I misunderstood what you currently have
in that case yeah, you only need one manage+settings
One per different type of service
In fact, do you need to publish your individual apps at all?
c

creamy-truck-92790

05/07/2021, 7:01 AM
ok. that makes sense then.
h

happy-kitchen-89482

05/07/2021, 7:01 AM
This might be much easier than I thought
I mistakenly thought you had 20 distinct services
c

creamy-truck-92790

05/07/2021, 7:02 AM
yes i do. because we have different clients for our ERP software and each of them use a different combination of our django apps / libaries
h

happy-kitchen-89482

05/07/2021, 7:02 AM
I see
c

creamy-truck-92790

05/07/2021, 7:02 AM
so on a client side we have one repository for each client which just installs the apps and have their customization. but i am completely fine with seperating them, at least for now away from the mono repo
h

happy-kitchen-89482

05/07/2021, 7:02 AM
So one thing to be aware of, and https://www.pantsbuild.org/docs/python-distributions explains this, is that you don't want the same shared library code published in multiple wheels
so it needs to be in its own wheel
that the per-app wheels depend on
I should go to sleep now, but let us know how it goes, happy to answer more questions
those doc pages have a lot of useful stuff
c

creamy-truck-92790

05/07/2021, 7:05 AM
I think I understand now how to continue. Thanks a lot for your help. Good night šŸ™‚
ā¤ļø 1
5 Views