https://pantsbuild.org/ logo
#general
Title
# general
g

gifted-agency-25998

11/08/2022, 9:38 AM
Hi all, I am currently evaluating pants for our fledgling MonoRepo and liking it so far 🙂 The main challenge I am facing is that we use flutter(frontend) and Pulumi(IaC), which are not (as far as I know) currently supported by pants. What is the most straightforward way to include them in the build process? For Pulumi, I would like it to detect
Pulumi.yaml
files and dependencies between different stacks (happy to describe this in the
BUILD
files) and perhaps do a
preview
on build and an
up
on publish. For flutter (I don't know enough about it) but would like to be able to integrate the build commands (
flutter pub get
,
flutter test
etc., in the relevant goals!) Any help with this would be much appreciated :)
h

happy-kitchen-89482

11/08/2022, 3:43 PM
A few folks have been asking about support for Pulumi recently. @faint-businessperson-86903 I believe you were working on something?
Ad for flutter - I also don’t know much about it, but I suspect creating a plugin for it would not be too hard since, as you say, it provides one underlying tool that already does the work we need, so it’s a matter of A) installing that tool (a preliminary flutter plugin could also assume it was available on the PATH), B) setting up the sandboxes to include the necessary files (a preliminary flutter plugin could ignore dep inference, and assume that you’ve manually added deps on the relevant targets) and C) running that tool
Someone would have to be motivated to do it
What goes into a flutter build? I assume .dart sources and lots of asset files?
Actually, I wonder if basic support could be added via
experimental_shell_command
today, cc @ancient-vegetable-10556
a

ancient-vegetable-10556

11/08/2022, 4:09 PM
If you’re interested in just running commands through pants, then
experimental_run_shell_command
is probably your best bet. That just lets you use Pants to run a given command, which you can string together with Pants’ goals using the Aliases feature. Actually using the smarter features inside pants is a lot harder
f

faint-businessperson-86903

11/08/2022, 4:14 PM
https://pantsbuild.slack.com/archives/C046T6T9U/p1667922183329779?thread_ts=1667900318.272189&channel=C046T6T9U&message_ts=1667922183.329779 My company has built a custom pants plugin for pulumi that takes advantage of some of the smarter features of pants, that we’ve been using for our non-frontend deployments for over a year now. I’ve shared the source to a few people here and would be happy to do so again. But because it’s something we’ve just been maintaining on the side there are some quirks, specifically around config and secret management. Also, as of today it’s only compatible with pants 2.12 (haven’t gotten around to updating yet, and the plug-in api is not yet stable)
g

gifted-agency-25998

11/08/2022, 4:19 PM
@faint-businessperson-86903, that's great - thank you. I would love to get access to the plugin and I'd be happy to share any back any changes we make. In terms of flutter, it has .dart, assets, as well as kotlin/java and swift (possibly event ObjC) for Android/IOS.
experimental_run_shell_command
seems like a reasonable stop-gap solution, and will try that out 🙂 Thank you all for your help 🙂
f

faint-businessperson-86903

11/08/2022, 4:48 PM
Happy to help, and would love to see any changes! Here is the zip:
You can install it by unzipping it into a source root (we have one just for plugins), and adding
pdlbuild.pulumi
to
backend_packages
in pants.toml. Feel free to reach out to me if you have any questions.
g

gifted-agency-25998

11/09/2022, 9:48 AM
I'll give it a shot (hopefully this week) and let you know - thank you so much for sharing 🙂
🔥 1
@faint-businessperson-86903, I finally got around to plugging this in and getting it to work, which was fairly straightforward. Thank you so much for putting this together and sharing it 🙂 I have a few question: • What is the LICENSE for the plugin? • Do you have plans for getting it on to github (or similar) to allow for collaboration? • I have a post-deploy step in some cases of syncing files to an s3 bucket. What would be simplest way to integrating this into the build flow?
f

faint-businessperson-86903

01/11/2023, 3:54 PM
Glad to hear it’s doing well for you! We also have an upgrade for this plugin making it compatible with pants 2.15.0rc1 going through testing/code review. Happy to share once it’s in use on our end. • I can grant this to you with a MIT license. I can share a copy with that embedded • We’ve been considering hosting it in our own repository and uploading it through pypi, or looking into upstreaming it into pants. We haven’t taken either step forward because it removes a lot of development steps to keep it in our monorepo, but having another company using it gives a good reason to do so 🙂 • You could of course do this inside Pulumi, with
aws.s3.BucketObject
resources or the synced folder provider. At that point it just becomes a natural part of your deployment flow. But with our build system we do have a way to pass in pre_deploy and post_deploy scripts (in the form of `./pants run`nable targets that happens from a higher-level internal tool that calls pants. We went this way because it didn’t feel appropriate to add the pre-deploy and post-deploy hooks at the target level.
g

gifted-agency-25998

01/11/2023, 4:57 PM
That's amazing on all fronts - thank you @faint-businessperson-86903 😄 For such modules, I have been toying with the idea of using git submodules, though since we are only just moving to mono-repos, I am unsure of how well it would work. We are currently using
BucketObject
to push files up, but I've read that it's a bit unwieldy when there are larger numbers of files as I believe it uploads them all again while
aws s3 sync
only uploads changed files. Having said that, I just realised that since the timestamps would be updated by the checkout (we run it in CI/CD), it'll upload them all anyway. The
pre_deploy
and
post_deploy
will definitely be useful for other actions though 🙂 I look forward to an updated package. I'll document the steps I had to take to get it working and ping it over to you as well to add in 🙂 Thanks again 😄
Hi @faint-businessperson-86903, I have a quick question if you don't mind 🙂 Our pulumi stacks have dependencies on other stacks. At first, I thought the dependencies field would cause the stacks to brought up in a specific folder. I now realise that it's more like python dependencies where the additional modules are included when
deploy
ing Is there a way to define order of deployments between multiple stacks, and ideally also the running of dependent stacks on changes to a dependee?
f

faint-businessperson-86903

01/30/2023, 5:36 PM
Hey @gifted-agency-25998! We haven’t thought about using the dependencies field for stacks to depend on other stacks, but that is a very interesting idea. Passing multiple
pulumi_program
targets into one
deploy
goal run isn’t a part of our workflow. But if we wanted it to work that way, it shouldn’t be that hard to pick out
pulumi_program
direct dependencies of other programs and use that info to organize a DAG.
That’s just one approach. To resolve this we follow a few practices internally: 1. Deploy (nearly) all changed stacks for a stage whenever a PR is merged, in a continuous-delivery-esque way. 2. Write dependee stacks such that they never assume the dependant stack is deployed at all. If an output X is needed to define Y set of resources, then return if X is unset, preventing Y from being a part of the stack. It’s annoying at times, but this way we get the intended result from deployments running a second time
g

gifted-agency-25998

01/30/2023, 5:47 PM
Thanks Tyler - that makes sense. We automate the deployments as well as part of the CI/CD Pipeline. I like the idea with 2nd option because that'll also allow preview to work when it has a dependency that we can't yet pick up (though the preview will be incomplete). It would also limit the the dependency to a maximum of 1 - otherwise, you'd need to run it multiple times. Do you automate the subsequent run, or do it manually. Depending on how painful the current situation gets and how much time I can muster, I'll maybe end up adding the dependency functionality into the plugin 🙂 In the meantime, we might opt for Option 1, but I guess it still presents the problem that the stacks might be deployed in the wrong order
f

faint-businessperson-86903

01/30/2023, 5:49 PM
Do you automate the subsequent run, or do it manually.
Automated runs are triggered regularly, so we usually let it happen organically, or manually if we are impatient 🙂
g

gifted-agency-25998

01/31/2023, 3:23 PM
makes sense - thank you 🙂
Hi @faint-businessperson-86903, Hope you are well 🙂 I am checking in to see if there are any updates on the pulumi plugin? No rush - just curious 🙂
4 Views