https://pantsbuild.org/ logo
t

thankful-elephant-10650

02/07/2023, 8:21 PM
Hello! We are a small Python team currently using Poetry in a monorepo-like setup (building and deploying multiple artifacts - Docker images, AWS Lambda functions). To build/deploy AWS Lambda and also manage infrastructure around these Lambda functions (S3, DynamoDB++) we're using CDK and are quite happy with it. I'm curious if anyone have had success with combining CDK and Pants. I see some mentions when searching, but these seems to point to specific issues. It would be nice to know if people were able to continue with CDK+Pants once they got over the hurdles mentioned here. @famous-river-94971 @chilly-holiday-77415 could you share your experience? šŸ™‚
šŸ‘‹ 2
b

broad-processor-92400

02/07/2023, 9:00 PM
We use pants+CDK for exactly docker images and lambda functions, but don't (yet) integrate them directly: we have CI run
./pants package --filter-tag-regex=stack1 ::
and then separately
cdk deploy ...
, with the CDK code referring to the appropriate
dist/path.to.target/whatever.zip
path for lambdas. We have multiple stacks deployed in separate CI jobs, so we set
tags=["stack1"]
(etc.) on the targets required for each stack, and use the
--filter-tag-regex
to only build what's required for each deploy.
t

thankful-elephant-10650

02/07/2023, 9:13 PM
Hi @broad-processor-92400, thanks for responding here! Right, this was my thinking as well - that it might be possible to run the two alongside each other and not necessarily having to invoke CDK from Pants (or the other way around).
@broad-processor-92400 Are you using the awslambda_python backend? https://www.pantsbuild.org/docs/awslambda-python
b

broad-processor-92400

02/07/2023, 9:30 PM
Yeah, we are (the one trick we had to get reliable behaviour was making sure we use `complete_platforms`: https://pantsbuild.slack.com/archives/C046T6T9U/p1673158061262749?thread_ts=1673153412.271249&cid=C046T6T9U)
c

chilly-holiday-77415

02/08/2023, 8:47 AM
currently Iā€™m pretty happy with what Iā€™m sure is a naive approach, but it worked well enough to cut the service I own into Pants full time - we cut the CDK code into a pex, and then reference ./dist/ā€¦ for lambda, also using
complete_platforms
to build lambdex. In CodePipeline (using CDK pipelines), my synth step now looks like (again, could probably be refined):
Copy code
"npm install -g cdk",
                    "./pants version",
                    "./pants package ::",
                    "cd infrastructure/main_service",
                    "cdk synth --app ../../dist/infrastructure.main_service/cdk_infra.pex -o $CODEBUILD_SRC_DIR/cdk.out",
I needed some help getting the BUILD file for the CDK app working:
Copy code
python_sources(
    name="cdklib",
    dependencies=[
        "//:reqs#aws-cdk-asset-awscli-v1",
        "//:reqs#jsii",
    ],
)

pex_binary(
    name="cdk_infra",
    entry_point="app.py",
    execution_mode="venv",
    venv_site_packages_copies=True,
)
f

famous-river-94971

02/08/2023, 8:37 PM
I've got this working and i'm relatively happy with it. Pants calls into cdk via the
experimental_run_shell
stuff.
This is all so much easier to do with
lerna
in TypeScript, but
pants
for Python was the next best thing I could find.
c

chilly-holiday-77415

02/09/2023, 9:21 AM
mind sharing an example Ben? Are you using
pants package
to synth?
f

famous-river-94971

02/09/2023, 3:15 PM
There's quite a bit of code and it's for a client, so I'd need to recreate it in a public repo. I currently don't have the bandwidth to commit to an example right away.
Here's a sample build file:
Copy code
python_sources(name="cdk_source", sources=["**/*.py"])
files(name="cdk_files", sources=["cdk.json", "cdk.context.json"])

experimental_run_shell_command(
    name="cdk_diff",
    tags=["cdk_diff"],
    command="../_scripts/cdk.sh diff --no-color",
    dependencies=[
        ":cdk_files",
        ":cdk_source",
    ],
    workdir="aws/projects/core_infrastructure",
)

experimental_run_shell_command(
    name="cdk_deploy",
    tags=["cdk_deploy"],
    command="../_scripts/cdk.sh deploy --progress events --require-approval never '**'",
    dependencies=[
        ":cdk_files",
        ":cdk_source",
    ],
    workdir="aws/projects/core_infrastructure",
)
And here's
_cdk.sh
Copy code
#! /bin/bash
set -e

SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
source "$SCRIPT_DIR/utils.sh"

source_virtualenv
set_aws_profile

cdk_version=$(find_aws_cdk_version)
npx -y aws-cdk@"$cdk_version" "$@"
And then this is my command in CI for diffing:
Copy code
./pants \
    --changed-since=origin/main \
    --changed-dependees=transitive \
    --filter-tag-regex='^cdk_diff$' \
    list | parallel \
      -L1 \
      --rpl '{project_name} $_ = s/(.*\/)(.*):.*/$2/r' \
      --tagstring '{project_name} |' \
      -P "$CONCURRENCY" \
      ./pants run
Side note: I really wish
pants
just had a
--parallel <n>
flag. Piping to
gnu parallel
adds a lot of visual complexity to the command.
Hopefully that gives at least a general idea of what I did?
t

thankful-elephant-10650

02/09/2023, 3:20 PM
Thank you very much! This definitely give us a good starting point!
šŸ™Œ 1
f

famous-river-94971

02/09/2023, 3:22 PM
This was a large monorepo, so the other thing I had to figure out was how to track "what had been deployed when". With
lerna
, they automatically add
git
tags to specify when things were deployed. So I could just say
--changed
and
lerna
would do all the heavy lifting. This doesn't exist in
pants
. So I basically recreated what lerna does manually in CI:
So then my
--changed-since
when deploying on
main
looks for the most recent
git tag
(sorting by timestamp) and uses that.
There are details in this thread: https://pantsbuild.slack.com/archives/C046T6T9U/p1673555303736359. If you're using Jenkins, there's a "last successful CI run" environment variable you can use for convenience. In my case, CircleCI doesn't have this.
So, in general, "does it work?". It does! Was it easy? Not really šŸ˜›
The CDK use case felt very off-road for what
pants
intends to do.
Also, having to use Python 3.9 for
pants
has been _rough_: https://github.com/pantsbuild/pants/issues/14111
The issue is closed but IIRC support for > 3.9 isn't coming in the RC, so it's going to be a while it seems
e

enough-analyst-54434

02/19/2023, 6:55 PM
In case it was not clear from that issue, if you use
scie-pants
for your
pants
binary (these install instructions: https://www.pantsbuild.org/v2.15/docs/installation), it works for any version of Pants all the way back to late 1.x and, although Pants still needs 3.8 or 3.9 you now don't have to care about that since
scie-pants
provides its own hermetic Python to run Pants with.
f

famous-river-94971

02/22/2023, 3:12 PM
Ah, so it is supported in 2.15 beta - I'll take a look at it. Thanks. I have my client set up with 2.14 stable right now.
e

enough-analyst-54434

02/22/2023, 3:59 PM
I'm sorry. I'm not being clear. The instructions to install Pants using
scie-pants
instead of the
./pants
script are only present in 2.15.x; however,
scie-pants
can in fact be used to run all versions of Pants. You can use it now for 2.14, 2.0, 1.26, etc. IOW,
scie-pants
is a drop in replacement for
./pants
- retroactively.
f

famous-river-94971

02/22/2023, 4:16 PM
Ohhh OK, I understand now. Thanks. That's great because I'd prefer to keep them on stable vs. beta for now šŸ™‚
Reviving an old thread, but I moved my client to
scie-pants
today. It's great to decouple the interpreter required for
pants
from my application! I suggested a few edits to the documentation and also left some additional information in this GitHub issue: https://github.com/pantsbuild/pants/issues/14111#issuecomment-1561759153. I'm impressed by how fast the interpreter download is šŸš— šŸ’Ø
šŸŽ‰ 1
6 Views