Hey folks - I'm evaluating using `pants` for a mon...
# general
Hey folks - I'm evaluating using
for a monorepo of AWS Cloud Development Kit (CDK), an infrastructure as code tool, projects. I've got a decent prototype set up, and it looks like the
flag is working great. It's identifying "targets" that should be redeployed when they change. I'm struggling to figure out the best way to run a shell command on targets that have changed. For instance, this command is properly identifying code that has changed:
Copy code
 ./pants --changed-since=HEAD --changed-dependees=transitive --filter-tag-regex='^cdk_app$' list
Now I'd like to run a single command in
cd aws/projects/project_1 && ./my-script.sh
), but I'm not sure how to best do this. In this case
would call out to the CDK binary to do the actual deployment 🧵
It wasn't clear to me if I should use one of these: https://www.pantsbuild.org/docs/run-shell-commands
Or even potentially define my own "goal" to run my CDK deployment?
TYIA for any ideas 🙂
It seems like
is giving me a list of files, but I really only care about "targets" (e.g., just the
There’s a “target” per source file, so in that case you may want to also filter on what target types you care about..
./pants peek
can be instructive here, revealing all the details.. 🙂
ah good tip
In my case, if any of the
files are changed, I know I need to run the CDK command in the top-level project folder
So maybe I have to do some shell-level filtering of the results to collapse that down.
Basically if any
file changes, I need to run a single command in
I’d probably look into having a
per project that depend on the code, and then list those targets for the changed-since, should turn them up when their deps change.
and with a macro to help reduce any repetitive boilerplate
ah, OK, so create a dependency on any code changing, interesting. I hadn't put that together.
(can be added later when you got it working)
I was a bit confused about when the shell command would actually run. I don't want it to run on developer machines when they're changing files, I want to just trigger it in CI when the files have changed
Ah, in this case it’s likely the
you want.. which you invoke explicitly with
./pants run …
👀 1
That's helpful! I was trying to figure out how to
some custom code.
N.b. there’s also support to run python sources directly..
./pants run src/python/main.py
Copy code
yarn lerna run --since '' --concurrency "$LERNA_CONCURRENCY" cdk -- deploy --progress events --require-approval never '**'
This is what I do for TypeScript projects using lerna
Yeah, the rub here is that you have to use the
executable installed on the CI runner. Then that binary reads the Python files where the infrastructure is defined.
Alright, thanks a ton Andreas, I'll give this a shot 🙂
cool, there’s so many ways to do this, comb through a bit and come back when you need more ideas/feedback 🙂
Will do, thank you again 🙂
we use this way, to figure out which pex target changed (under a specific folder which has all of our deployable pex targets):
Copy code
changed_targets=`./pants --changed-since=<< pipeline.git.base_revision >> --changed-dependees=transitive --filter-target-type=pex_binary --filter-address-regex="src/python/toolchain/service.*" list
this is part of the script, the list of pex targets that need to be deployed is assigned into
variable in the bash script (in CI)
👍 1
This looks pretty promising:
Copy code
commit 29ff052b413bbf497192973f28f776c6344cb512 (HEAD -> pants)
Author: Ben Limmer <ben@benlimmer.com>
Date:   Wed Jan 4 14:24:30 2023 -0700

    try adding a script

diff --git a/aws/projects/project_1/BUILD b/aws/projects/project_1/BUILD
index a15e303..1dcab2b 100644
--- a/aws/projects/project_1/BUILD
+++ b/aws/projects/project_1/BUILD
@@ -1,3 +1,9 @@
-  tags=["cdk_app"]
+  name="cdk",
+  tags=["cdk"],
+  command="../../scripts/cdk-deploy.sh",
+  dependencies=["aws/projects/project_1:project_1"],
+  workdir="aws/projects/project_1",
diff --git a/aws/scripts/BUILD b/aws/scripts/BUILD
new file mode 100644
index 0000000..6c95f66
--- /dev/null
+++ b/aws/scripts/BUILD
@@ -0,0 +1 @@
diff --git a/aws/scripts/cdk-deploy.sh b/aws/scripts/cdk-deploy.sh
new file mode 100755
index 0000000..e7a3bbc
--- /dev/null
+++ b/aws/scripts/cdk-deploy.sh
@@ -0,0 +1,4 @@
+#! /bin/bash
+set -e
+echo "hello from $(basename "$(pwd)")"
diff --git a/pants.toml b/pants.toml
index b0ce38c..2f954eb 100644
--- a/pants.toml
+++ b/pants.toml
@@ -3,6 +3,7 @@ pants_version = "2.14.0"
 backend_packages = [
+  "pants.backend.shell",

Then if I make a change in my
source files (either directly or transitively) I get this:
Copy code
 ./pants --changed-since=HEAD --changed-dependees=transitive --filter-tag-regex='^cdk$' run
hello from project_1
which is what I expect 🎉
And with multiple changes, e.g.,
Copy code
 ./pants --changed-since=HEAD --changed-dependees=transitive --filter-tag-regex='^cdk$' list
It looks like I could use xargs like this to execute
on each:
Copy code
 ./pants --changed-since=HEAD --changed-dependees=transitive --filter-tag-regex='^cdk$' list | xargs -L1 ./pants run
hello from project_1
hello from project_2
I'll have to do some thinking about how to introduce concurrency to this, but I feel like this is a good starting solution!
Next thing I'm trying to figure out - I need all the pants-specific Python context (e.g., module linking, Python path, etc.) to be active when I run my script.
Looks like I can run
pants export
and source the virtualenv. Then export
to get module resolution working.
Hey Ben - I’m also experimenting with pants + CDK, although my monorepo only has a single CDK infrastructure and many python projects (lambdas etc) for now. Sounds like you’re making good progress!
👍 1