https://pantsbuild.org/ logo
Join Slack
Powered by
Hey folks - I'm evaluating using `pants` for a mon...
# general
f
Hey folks - I'm evaluating using 
pants
for a monorepo of AWS Cloud Development Kit (CDK), an infrastructure as code tool, projects. I've got a decent prototype set up, and it looks like the 
--changed-since
flag is working great. It's identifying "targets" that should be redeployed when they change. I'm struggling to figure out the best way to run a shell command on targets that have changed. For instance, this command is properly identifying code that has changed:
Copy code
 ./pants --changed-since=HEAD --changed-dependees=transitive --filter-tag-regex='^cdk_app$' list
aws/projects/project_1:project_1
aws/projects/project_1/app.py
aws/projects/project_1/stack.py
Now I'd like to run a single command in 
project_1
(e.g., 
cd aws/projects/project_1 && ./my-script.sh
), but I'm not sure how to best do this. In this case 
my-script
would call out to the CDK binary to do the actual deployment 🧵
It wasn't clear to me if I should use one of these: https://www.pantsbuild.org/docs/run-shell-commands
Or even potentially define my own "goal" to run my CDK deployment?
TYIA for any ideas 🙂
It seems like 
list
is giving me a list of files, but I really only care about "targets" (e.g., just the 
project_1
path)
c
There’s a “target” per source file, so in that case you may want to also filter on what target types you care about..
./pants peek
can be instructive here, revealing all the details.. 🙂
f
ah good tip
In my case, if any of the 
python_source
files are changed, I know I need to run the CDK command in the top-level project folder
So maybe I have to do some shell-level filtering of the results to collapse that down.
Basically if any 
aws/projects/project_1/**/*.py
file changes, I need to run a single command in 
aws/projects/project_1
c
I’d probably look into having a 
experimental_shell_command
per project that depend on the code, and then list those targets for the changed-since, should turn them up when their deps change.
and with a macro to help reduce any repetitive boilerplate
f
ah, OK, so create a dependency on any code changing, interesting. I hadn't put that together.
c
(can be added later when you got it working)
f
I was a bit confused about when the shell command would actually run. I don't want it to run on developer machines when they're changing files, I want to just trigger it in CI when the files have changed
c
Ah, in this case it’s likely the 
experimental_run_shell_command
you want.. which you invoke explicitly with 
./pants run …
👀 1
f
That's helpful! I was trying to figure out how to 
run
some custom code.
c
N.b. there’s also support to run python sources directly.. 
./pants run src/python/main.py
…
f
Copy code
yarn lerna run --since '' --concurrency "$LERNA_CONCURRENCY" cdk -- deploy --progress events --require-approval never '**'
This is what I do for TypeScript projects using lerna
Yeah, the rub here is that you have to use the 
cdk
executable installed on the CI runner. Then that binary reads the Python files where the infrastructure is defined.
Alright, thanks a ton Andreas, I'll give this a shot 🙂
c
cool, there’s so many ways to do this, comb through a bit and come back when you need more ideas/feedback 🙂
f
Will do, thank you again 🙂
p
we use this way, to figure out which pex target changed (under a specific folder which has all of our deployable pex targets):
Copy code
changed_targets=`./pants --changed-since=<< pipeline.git.base_revision >> --changed-dependees=transitive --filter-target-type=pex_binary --filter-address-regex="src/python/toolchain/service.*" list
this is part of the script, the list of pex targets that need to be deployed is assigned into 
changed_targets
variable in the bash script (in CI)
👍 1
f
This looks pretty promising:
Copy code
commit 29ff052b413bbf497192973f28f776c6344cb512 (HEAD -> pants)
Author: Ben Limmer <ben@benlimmer.com>
Date:   Wed Jan 4 14:24:30 2023 -0700

    try adding a script

diff --git a/aws/projects/project_1/BUILD b/aws/projects/project_1/BUILD
index a15e303..1dcab2b 100644
--- a/aws/projects/project_1/BUILD
+++ b/aws/projects/project_1/BUILD
@@ -1,3 +1,9 @@
-python_sources(
-  tags=["cdk_app"]
+python_sources()
+
+experimental_run_shell_command(
+  name="cdk",
+  tags=["cdk"],
+  command="../../scripts/cdk-deploy.sh",
+  dependencies=["aws/projects/project_1:project_1"],
+  workdir="aws/projects/project_1",
 )
diff --git a/aws/scripts/BUILD b/aws/scripts/BUILD
new file mode 100644
index 0000000..6c95f66
--- /dev/null
+++ b/aws/scripts/BUILD
@@ -0,0 +1 @@
+shell_sources()
diff --git a/aws/scripts/cdk-deploy.sh b/aws/scripts/cdk-deploy.sh
new file mode 100755
index 0000000..e7a3bbc
--- /dev/null
+++ b/aws/scripts/cdk-deploy.sh
@@ -0,0 +1,4 @@
+#! /bin/bash
+set -e
+
+echo "hello from $(basename "$(pwd)")"
diff --git a/pants.toml b/pants.toml
index b0ce38c..2f954eb 100644
--- a/pants.toml
+++ b/pants.toml
@@ -3,6 +3,7 @@ pants_version = "2.14.0"
 backend_packages = [
   "pants.backend.python",
   "pants.backend.python.lint.black",
+  "pants.backend.shell",
 ]

 [source]
Then if I make a change in my 
project_1
source files (either directly or transitively) I get this:
Copy code
 ./pants --changed-since=HEAD --changed-dependees=transitive --filter-tag-regex='^cdk$' run
hello from project_1
which is what I expect 🎉
And with multiple changes, e.g.,
Copy code
 ./pants --changed-since=HEAD --changed-dependees=transitive --filter-tag-regex='^cdk$' list
aws/projects/project_1:cdk
aws/projects/project_2:cdk
It looks like I could use xargs like this to execute 
run
on each:
Copy code
 ./pants --changed-since=HEAD --changed-dependees=transitive --filter-tag-regex='^cdk$' list | xargs -L1 ./pants run
hello from project_1
hello from project_2
I'll have to do some thinking about how to introduce concurrency to this, but I feel like this is a good starting solution!
Next thing I'm trying to figure out - I need all the pants-specific Python context (e.g., module linking, Python path, etc.) to be active when I run my script.
Looks like I can run 
pants export
and source the virtualenv. Then export 
PYTHONPATH
to get module resolution working.
c
Hey Ben - I’m also experimenting with pants + CDK, although my monorepo only has a single CDK infrastructure and many python projects (lambdas etc) for now. Sounds like you’re making good progress!
👍 1
6 Views
Open in Slack
PreviousNext
Powered by