https://pantsbuild.org/ logo
Join Slack
Powered by
Hey all, so I just got approval to open-source the...
# development
r
Hey all, so I just got approval to open-source the pants dbt and dagster backends I've been building at work. Should I look to open-source these as plugins in their own repo or would there be interest in having these in pants mainline? (IMO it doesn't really make sense for the dagster one but I could see the case for dbt) If so, should I create a feature request for it?
šŸŽ‰ 4
šŸŖ© 2
ā­ 5
šŸ¦• 3
g
Nice work! I've been pushing for less plugins in the pants repo since it delegates maintainership to users more clearly. Also unties the release cycle from the Pants release cycle, allowing faster feature shipping etc.
I've been thinking a bit about something like bazel-contrib, which is a community governed group focused on bazel rules bundles... But I'm also quite happy to hack on my own. šŸ˜›
b
I think there's tension (in some ways, the usual mono-repo vs. poly-repo tension): ā€¢ in-repo plugins automatically work with the version of Pants they're packaged with, since they're updated for breaking changes and Pants' plugin API is definitely unstable ā€¢ out-of-repo plugins have the benefits Tom mentions... assuming that someone is around to keep them current šŸ¤·
g
If noone is around to keep them current... either noone cares or someone has to step up and do it.
šŸ‘ 1
I realize it is sort of a hardline approach but for example, the committed-and-forgotten swift backend with one meaningful commit ever... is it working? does anyone use it? will it work in X amount of time? who knows how to fix it when it doesn't?
šŸ‘ 2
c
Everyone has a slightly different inclination. Tom & Huon outlined the standard tradeoffs. Please take this in the spirit of what makes the most sense for the plugins; they deserve a hearty welcome wherever they end up! I think most of the current out of tree plugins are rather tinkering oriented and fly a bit under the radar, but I think at the least a list on the Pants docs page would make a lot of sense. IMO if your intuition is that Dagster makes the most sense out of tree that's probably right. I could see dbt interacting with various sql linters/tooling in a way where being in-tree could be smoother. This is hand waving though as I'm only guessing at what the plugin does! (As someone working on converting a dbt-using-project to Pants right now I'm excited to learn)
(If possible, matching matching the Apache License 2.0 Pants itself uses would be the biggest hard-to-change-later decision to keep all these options open in the future.)
h
This is awesome, thanks for open-sourcing these!
The main drawback of having them outside the Pants repo is that the plugin API is not yet stable, and so when it changes it may break your plugins, and we won't know.
But while not officially stabilized, the API is quite stable in practice, and I think it would be good for us to dip our toe in the water of having some plugins outside the Pants repo
I guess one thing we should add is a way for plugins to declare which Pants versions they have been proven to work with
But I wouldn't wait for that
in the short term, would you be willing to set up CI for your plugins that tests against latest dev Pants, so you get early warning of breakage?
g
^ This is what I do; I run a matrix test vs "previous", "current" and "next" pants. I've not found many bad breakages, but it does help me pre-fix changes in future versions (by branching on pants version).
r
I also need to figure out the best way to break these out of our work monorepo. There's some hacky stuff in both backends that's pretty specific to our use case that could probably be improved/generalized in an open-source solution.
@curved-manchester-66006 Would definitely be interested in your take on how I've set up the targets and such. I set up separate targets for models, macros, tests, etc. but given that afaik you must define where this stuff lives in the 
dbt_project.yml
part of me wonders if there should just be a single target for an entire project that generates all of the targets for its nodes.
...which is actually not possible now that I look at it, since each 
TargetGenerator
can only generate a single type.
c
FWIW My colleagues have decided to take our One Big DBT Project and refactor it into multiple interdependent projects, so my interest here has increased :-)
r
Wanted to post an update here because unfortunately I've been impacted by layoffs and no longer have access to the repo these plugins were in. That being said, I probably could recreate them from scratch and I'm about to have nothing but time on my hands for a little while (especially if the shit weather continues here in Philly) so maybe watch this space.
šŸ’œ 1
c
Regarding a target generator only generating a single type is not accurate. You can generate any targets in a hopscotch mix as you wish. Might not be the best idea, but could be valid occasionally I think.
h
Sorry to hear that @ripe-gigabyte-88964! Hopefully you can take some time to recharge (and hack on Pants... šŸ˜‰ )
Several people in the Pants community have found job opportunities on here, so when the time is right it might be worth broadcasting here
a
@curved-television-6568 Do you have any examples of this?
Maybe I can create a generic superclass or just set the 
generated_target_cls
to 
Target
.
c
It's for 
python_requirements
which in addition to the generated 
python_requirement
targets, it also creates the internal 
_generator_sources_helper
target.
2 Views
Open in Slack
PreviousNext
Powered by