https://pantsbuild.org/ logo
Join Slack
Powered by
Hello! We are a small Python team currently using ...
# general
t
Hello! We are a small Python team currently using Poetry in a monorepo-like setup (building and deploying multiple artifacts - Docker images, AWS Lambda functions). To build/deploy AWS Lambda and also manage infrastructure around these Lambda functions (S3, DynamoDB++) we're using CDK and are quite happy with it. I'm curious if anyone have had success with combining CDK and Pants. I see some mentions when searching, but these seems to point to specific issues. It would be nice to know if people were able to continue with CDK+Pants once they got over the hurdles mentioned here. @famous-river-94971 @chilly-holiday-77415 could you share your experience? šŸ™‚
šŸ‘‹ 2
b
We use pants+CDK for exactly docker images and lambda functions, but don't (yet) integrate them directly: we have CI run 
./pants package --filter-tag-regex=stack1 ::
and then separately 
cdk deploy ...
, with the CDK code referring to the appropriate 
dist/path.to.target/whatever.zip
path for lambdas. We have multiple stacks deployed in separate CI jobs, so we set 
tags=["stack1"]
(etc.) on the targets required for each stack, and use the 
--filter-tag-regex
to only build what's required for each deploy.
t
Hi @broad-processor-92400, thanks for responding here! Right, this was my thinking as well - that it might be possible to run the two alongside each other and not necessarily having to invoke CDK from Pants (or the other way around).
@broad-processor-92400 Are you using the awslambda_python backend? https://www.pantsbuild.org/docs/awslambda-python
b
Yeah, we are (the one trick we had to get reliable behaviour was making sure we use `complete_platforms`: https://pantsbuild.slack.com/archives/C046T6T9U/p1673158061262749?thread_ts=1673153412.271249&cid=C046T6T9U)
c
currently Iā€™m pretty happy with what Iā€™m sure is a naive approach, but it worked well enough to cut the service I own into Pants full time - we cut the CDK code into a pex, and then reference ./dist/ā€¦ for lambda, also using 
complete_platforms
to build lambdex. In CodePipeline (using CDK pipelines), my synth step now looks like (again, could probably be refined):
Copy code
"npm install -g cdk",
                    "./pants version",
                    "./pants package ::",
                    "cd infrastructure/main_service",
                    "cdk synth --app ../../dist/infrastructure.main_service/cdk_infra.pex -o $CODEBUILD_SRC_DIR/cdk.out",
I needed some help getting the BUILD file for the CDK app working:
Copy code
python_sources(
    name="cdklib",
    dependencies=[
        "//:reqs#aws-cdk-asset-awscli-v1",
        "//:reqs#jsii",
    ],
)

pex_binary(
    name="cdk_infra",
    entry_point="app.py",
    execution_mode="venv",
    venv_site_packages_copies=True,
)
f
I've got this working and i'm relatively happy with it. Pants calls into cdk via the 
experimental_run_shell
stuff.
This is all so much easier to do with 
lerna
in TypeScript, but 
pants
for Python was the next best thing I could find.
c
mind sharing an example Ben? Are you using 
pants package
to synth?
f
There's quite a bit of code and it's for a client, so I'd need to recreate it in a public repo. I currently don't have the bandwidth to commit to an example right away.
Here's a sample build file:
Copy code
python_sources(name="cdk_source", sources=["**/*.py"])
files(name="cdk_files", sources=["cdk.json", "cdk.context.json"])

experimental_run_shell_command(
    name="cdk_diff",
    tags=["cdk_diff"],
    command="../_scripts/cdk.sh diff --no-color",
    dependencies=[
        ":cdk_files",
        ":cdk_source",
    ],
    workdir="aws/projects/core_infrastructure",
)

experimental_run_shell_command(
    name="cdk_deploy",
    tags=["cdk_deploy"],
    command="../_scripts/cdk.sh deploy --progress events --require-approval never '**'",
    dependencies=[
        ":cdk_files",
        ":cdk_source",
    ],
    workdir="aws/projects/core_infrastructure",
)
And here's 
_cdk.sh
Copy code
#! /bin/bash
set -e

SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
source "$SCRIPT_DIR/utils.sh"

source_virtualenv
set_aws_profile

cdk_version=$(find_aws_cdk_version)
npx -y aws-cdk@"$cdk_version" "$@"
And then this is my command in CI for diffing:
Copy code
./pants \
    --changed-since=origin/main \
    --changed-dependees=transitive \
    --filter-tag-regex='^cdk_diff$' \
    list | parallel \
      -L1 \
      --rpl '{project_name} $_ = s/(.*\/)(.*):.*/$2/r' \
      --tagstring '{project_name} |' \
      -P "$CONCURRENCY" \
      ./pants run
Side note: I really wish 
pants
just had a 
--parallel <n>
flag. Piping to 
gnu parallel
adds a lot of visual complexity to the command.
Hopefully that gives at least a general idea of what I did?
t
Thank you very much! This definitely give us a good starting point!
šŸ™Œ 1
f
This was a large monorepo, so the other thing I had to figure out was how to track "what had been deployed when". With 
lerna
, they automatically add 
git
tags to specify when things were deployed. So I could just say 
--changed
and 
lerna
would do all the heavy lifting. This doesn't exist in 
pants
. So I basically recreated what lerna does manually in CI:
So then my 
--changed-since
when deploying on 
main
looks for the most recent 
git tag
(sorting by timestamp) and uses that.
There are details in this thread: https://pantsbuild.slack.com/archives/C046T6T9U/p1673555303736359. If you're using Jenkins, there's a "last successful CI run" environment variable you can use for convenience. In my case, CircleCI doesn't have this.
So, in general, "does it work?". It does! Was it easy? Not really šŸ˜›
The CDK use case felt very off-road for what 
pants
intends to do.
Also, having to use Python 3.9 for 
pants
has been _rough_: https://github.com/pantsbuild/pants/issues/14111
The issue is closed but IIRC support for > 3.9 isn't coming in the RC, so it's going to be a while it seems
e
In case it was not clear from that issue, if you use 
scie-pants
for your 
pants
binary (these install instructions: https://www.pantsbuild.org/v2.15/docs/installation), it works for any version of Pants all the way back to late 1.x and, although Pants still needs 3.8 or 3.9 you now don't have to care about that since 
scie-pants
provides its own hermetic Python to run Pants with.
f
Ah, so it is supported in 2.15 beta - I'll take a look at it. Thanks. I have my client set up with 2.14 stable right now.
e
I'm sorry. I'm not being clear. The instructions to install Pants using 
scie-pants
instead of the 
./pants
script are only present in 2.15.x; however, 
scie-pants
can in fact be used to run all versions of Pants. You can use it now for 2.14, 2.0, 1.26, etc. IOW, 
scie-pants
is a drop in replacement for 
./pants
- retroactively.
f
Ohhh OK, I understand now. Thanks. That's great because I'd prefer to keep them on stable vs. beta for now šŸ™‚
Reviving an old thread, but I moved my client to 
scie-pants
today. It's great to decouple the interpreter required for 
pants
from my application! I suggested a few edits to the documentation and also left some additional information in this GitHub issue: https://github.com/pantsbuild/pants/issues/14111#issuecomment-1561759153. I'm impressed by how fast the interpreter download is šŸš— šŸ’Ø
šŸŽ‰ 1
6 Views
Open in Slack
PreviousNext
Powered by