https://pantsbuild.org/ logo
b

breezy-mouse-20493

04/24/2023, 11:15 PM
Hi everyone, looking for advice in porting projects under separate repos with setup.py files into a monorepo. First question: Do I need to port the setup.py files into pyproject.toml files? Does Pants require pyproject.toml?
e

enough-analyst-54434

04/24/2023, 11:28 PM
It does not, but it also treats these files as completely secondary. You use them to publish a distribution and nothing else generally. Pants directly works with your project python files and understands their dependencies (both 1st and 3rd party) without looking at those sorts of files at all.
Do you use such files for publishing for external consumption (~PyPI) or just because that's how the typical Python project rolls?
b

breezy-mouse-20493

04/24/2023, 11:29 PM
Yes, we publish to an internal PyPi server.
e

enough-analyst-54434

04/24/2023, 11:30 PM
For what purpose though? When all projects are in a monorepo will there be a need for that?
b

breezy-mouse-20493

04/24/2023, 11:30 PM
In the monorepo we want to have certain packages published at their own cadence.
Is this supported?
e

enough-analyst-54434

04/24/2023, 11:31 PM
Ouch, ok. Yes, but it's generally not tenable / sort of defeats the point of being in the monorepo.
b

breezy-mouse-20493

04/24/2023, 11:31 PM
Okay.
e

enough-analyst-54434

04/24/2023, 11:32 PM
Is this a social problem or a real technical concern?
b

breezy-mouse-20493

04/24/2023, 11:33 PM
The packages that should be built on their own cadence -- should these be separate "pants monorepos"? Social vs technical: More social, I think.
(but this is a good question which I will have to think about)
e

enough-analyst-54434

04/24/2023, 11:34 PM
Yeah. I've spent a good deal of time in monorepos where it was basically social. It's unfortunate but a common real problem fwict.
Personally I'd gate entry into the monorepo on that. If you want to allow people to dev at their own pace, I'd personally say your on your own.
b

breezy-mouse-20493

04/24/2023, 11:35 PM
Yes. Since I have you here, what are your thoughts on the "hello world" Pants project on github for Python? It's a small project, but is one takeaway that Pants can/should be used for small, single-module repos?
e

enough-analyst-54434

04/24/2023, 11:36 PM
Pants should have ~0 to do with size or structure of a repo. Where it does it's broken.
👌 1
It's true it grew from a monorepo, but it should no longer really care.
b

breezy-mouse-20493

04/24/2023, 11:38 PM
Regarding setup.py/pyproject.toml: Neither of these are need because Pants will infer dependencies, correct? This is interesting. So then the entry point for developers into the Pants system is just pants.toml?
e

enough-analyst-54434

04/24/2023, 11:38 PM
Correct. And then, for Python, somewhere to write down 3rd party constraints. That can be requirements.txt or several other forms.
b

breezy-mouse-20493

04/24/2023, 11:39 PM
Great. Thanks @average-flag-94768. I'll go back to the docs now that a few things are cleared up. Thanks for your help.
h

happy-kitchen-89482

04/25/2023, 12:04 AM
To clarify, Pants will happily build wheels and publish them to PyPI (or an internal server). It can use your existing setup.py for this or generate them. John's point is that you should be very sure you need to do this. Often it is done because Python has not historically had a good story for managing multiple units/packages in a single repo, but Pants provides that. So if you must publish because you have consumers outside the repo, that must stay outside the repo then do so, of course. But it'll be much easier for you not to whenever you don't have to.
b

breezy-mouse-20493

04/25/2023, 12:05 AM
I'm watching your Youtube video on this now @happy-kitchen-89482 🙂
🎥 1
The one from the Earthly channel.
I think that because of the way our org is set up, we will likely have to support the "internal third party" scenario. I'll have to talk this over with the teams.
h

happy-kitchen-89482

04/25/2023, 12:08 AM
At least try and minimize the number of wheels you need to publish, it is such a headache compared to deploying a fully self-contained binary built at some SHA
b

breezy-mouse-20493

04/25/2023, 12:08 AM
Agreed.
@happy-kitchen-89482 We have quite a few internal users of tools that will end up importing modules from the monorepo -- is this a good build target for Pants? I see a lot of talk around the Pex format. What's the story around pip installable modules in e.g. a company-wide Jupyter environment? If there is a docs page that I haven't yet seen, I'd gladly look at that if you want to point me there.
h

happy-kitchen-89482

04/25/2023, 12:18 AM
Pex is that standalone single-file binary format, it is great for deploying services etc
But it's not suitable for importable library dependencies, that's what wheels are for
so it sounds like you do have to build and publish some wheels from the monorepo, and that's fine
This is the documentation for that: https://www.pantsbuild.org/docs/python-distributions
That is how you'd set up and build anything that needs to be pip-installable outside the repo
👌 1
s

swift-river-73520

04/25/2023, 12:20 AM
you could consider deploying pex files to your instances that host your jupyter environment and build virtual environment from them, I've been doing that as a really clean way to basically ship an environment to a remote worker if I need it to be extendable
well I guess I don't know if that's considered "clean" lol, but it's a really nice way to ensure that the remote environment is the same as what you built and not have to pip install anything
b

breezy-mouse-20493

04/25/2023, 12:21 AM
@swift-river-73520 that's a compelling argument and I'll look into it. Thank you.
Will have to work with our users to get them onto the Pex format. Compared to what we have now ("venv hell") this should be easy to promote to the team.
2 Views