https://pantsbuild.org/ logo
#general
Title
# general
f

famous-river-94971

01/04/2023, 8:50 PM
Hey folks - I'm evaluating using
pants
for a monorepo of AWS Cloud Development Kit (CDK), an infrastructure as code tool, projects. I've got a decent prototype set up, and it looks like the
--changed-since
flag is working great. It's identifying "targets" that should be redeployed when they change. I'm struggling to figure out the best way to run a shell command on targets that have changed. For instance, this command is properly identifying code that has changed:
Copy code
 ./pants --changed-since=HEAD --changed-dependees=transitive --filter-tag-regex='^cdk_app$' list
aws/projects/project_1:project_1
aws/projects/project_1/app.py
aws/projects/project_1/stack.py
Now I'd like to run a single command in
project_1
(e.g.,
cd aws/projects/project_1 && ./my-script.sh
), but I'm not sure how to best do this. In this case
my-script
would call out to the CDK binary to do the actual deployment 🧵
It wasn't clear to me if I should use one of these: https://www.pantsbuild.org/docs/run-shell-commands
Or even potentially define my own "goal" to run my CDK deployment?
TYIA for any ideas 🙂
It seems like
list
is giving me a list of files, but I really only care about "targets" (e.g., just the
project_1
path)
c

curved-television-6568

01/04/2023, 9:06 PM
There’s a “target” per source file, so in that case you may want to also filter on what target types you care about..
./pants peek
can be instructive here, revealing all the details.. 🙂
f

famous-river-94971

01/04/2023, 9:06 PM
ah good tip
In my case, if any of the
python_source
files are changed, I know I need to run the CDK command in the top-level project folder
So maybe I have to do some shell-level filtering of the results to collapse that down.
Basically if any
aws/projects/project_1/**/*.py
file changes, I need to run a single command in
aws/projects/project_1
c

curved-television-6568

01/04/2023, 9:08 PM
I’d probably look into having a
experimental_shell_command
per project that depend on the code, and then list those targets for the changed-since, should turn them up when their deps change.
and with a macro to help reduce any repetitive boilerplate
f

famous-river-94971

01/04/2023, 9:09 PM
ah, OK, so create a dependency on any code changing, interesting. I hadn't put that together.
c

curved-television-6568

01/04/2023, 9:09 PM
(can be added later when you got it working)
f

famous-river-94971

01/04/2023, 9:10 PM
I was a bit confused about when the shell command would actually run. I don't want it to run on developer machines when they're changing files, I want to just trigger it in CI when the files have changed
c

curved-television-6568

01/04/2023, 9:10 PM
Ah, in this case it’s likely the
experimental_run_shell_command
you want.. which you invoke explicitly with
./pants run …
👀 1
f

famous-river-94971

01/04/2023, 9:11 PM
That's helpful! I was trying to figure out how to
run
some custom code.
c

curved-television-6568

01/04/2023, 9:11 PM
N.b. there’s also support to run python sources directly..
./pants run src/python/main.py
f

famous-river-94971

01/04/2023, 9:11 PM
Copy code
yarn lerna run --since '' --concurrency "$LERNA_CONCURRENCY" cdk -- deploy --progress events --require-approval never '**'
This is what I do for TypeScript projects using lerna
Yeah, the rub here is that you have to use the
cdk
executable installed on the CI runner. Then that binary reads the Python files where the infrastructure is defined.
Alright, thanks a ton Andreas, I'll give this a shot 🙂
c

curved-television-6568

01/04/2023, 9:13 PM
cool, there’s so many ways to do this, comb through a bit and come back when you need more ideas/feedback 🙂
f

famous-river-94971

01/04/2023, 9:13 PM
Will do, thank you again 🙂
p

polite-garden-50641

01/04/2023, 9:24 PM
we use this way, to figure out which pex target changed (under a specific folder which has all of our deployable pex targets):
Copy code
changed_targets=`./pants --changed-since=<< pipeline.git.base_revision >> --changed-dependees=transitive --filter-target-type=pex_binary --filter-address-regex="src/python/toolchain/service.*" list
this is part of the script, the list of pex targets that need to be deployed is assigned into
changed_targets
variable in the bash script (in CI)
👍 1
f

famous-river-94971

01/04/2023, 9:36 PM
This looks pretty promising:
Copy code
commit 29ff052b413bbf497192973f28f776c6344cb512 (HEAD -> pants)
Author: Ben Limmer <ben@benlimmer.com>
Date:   Wed Jan 4 14:24:30 2023 -0700

    try adding a script

diff --git a/aws/projects/project_1/BUILD b/aws/projects/project_1/BUILD
index a15e303..1dcab2b 100644
--- a/aws/projects/project_1/BUILD
+++ b/aws/projects/project_1/BUILD
@@ -1,3 +1,9 @@
-python_sources(
-  tags=["cdk_app"]
+python_sources()
+
+experimental_run_shell_command(
+  name="cdk",
+  tags=["cdk"],
+  command="../../scripts/cdk-deploy.sh",
+  dependencies=["aws/projects/project_1:project_1"],
+  workdir="aws/projects/project_1",
 )
diff --git a/aws/scripts/BUILD b/aws/scripts/BUILD
new file mode 100644
index 0000000..6c95f66
--- /dev/null
+++ b/aws/scripts/BUILD
@@ -0,0 +1 @@
+shell_sources()
diff --git a/aws/scripts/cdk-deploy.sh b/aws/scripts/cdk-deploy.sh
new file mode 100755
index 0000000..e7a3bbc
--- /dev/null
+++ b/aws/scripts/cdk-deploy.sh
@@ -0,0 +1,4 @@
+#! /bin/bash
+set -e
+
+echo "hello from $(basename "$(pwd)")"
diff --git a/pants.toml b/pants.toml
index b0ce38c..2f954eb 100644
--- a/pants.toml
+++ b/pants.toml
@@ -3,6 +3,7 @@ pants_version = "2.14.0"
 backend_packages = [
   "pants.backend.python",
   "pants.backend.python.lint.black",
+  "pants.backend.shell",
 ]

 [source]
Then if I make a change in my
project_1
source files (either directly or transitively) I get this:
Copy code
 ./pants --changed-since=HEAD --changed-dependees=transitive --filter-tag-regex='^cdk$' run
hello from project_1
which is what I expect 🎉
And with multiple changes, e.g.,
Copy code
 ./pants --changed-since=HEAD --changed-dependees=transitive --filter-tag-regex='^cdk$' list
aws/projects/project_1:cdk
aws/projects/project_2:cdk
It looks like I could use xargs like this to execute
run
on each:
Copy code
 ./pants --changed-since=HEAD --changed-dependees=transitive --filter-tag-regex='^cdk$' list | xargs -L1 ./pants run
hello from project_1
hello from project_2
I'll have to do some thinking about how to introduce concurrency to this, but I feel like this is a good starting solution!
Next thing I'm trying to figure out - I need all the pants-specific Python context (e.g., module linking, Python path, etc.) to be active when I run my script.
Looks like I can run
pants export
and source the virtualenv. Then export
PYTHONPATH
to get module resolution working.
c

chilly-holiday-77415

01/05/2023, 9:46 AM
Hey Ben - I’m also experimenting with pants + CDK, although my monorepo only has a single CDK infrastructure and many python projects (lambdas etc) for now. Sounds like you’re making good progress!
👍 1
6 Views