Just wanted to share some neatness I found: I have a few different repos that utilize

python_aws_lambda_function

but I could never find a decent way to integrate the build target with

terraform_module

until I recently tried it with

shell_command

as a intermediary. Here is my

src/terraform/BUILD

file:

terraform_module(dependencies=[":lambda_zip"])

terraform_deployment(name="infra", root_module=":terraform")

shell_command(
    name="lambda_zip",
    command="[ -f 'src.python/lambda.zip' ] || { exit 1; }",
    execution_dependencies=["src/python:lambda"],
    output_files=["src.python/lambda.zip"],
    workdir="/",
)

and I read the

lambda.zip

using:

data "local_file" "lambda_package" {
  filename = "../../src.python/lambda.zip"
}

Hey there folks. I have a potentially very pants novice question about why an intermediate shell command is needed to use the outputted package of a

python_aws_lambda_function

. I was trying out terraform deployment using pants for the first time today and my intuitive setup was something like this:

# Lambda package setup
python_sources(name="python", resolve="inference")
python_aws_lambda_function(
    name="terraform_pants_lambda",
    dependencies=[":python"],
    handler="hello.py:handler",
    runtime="python3.11",
    resolve="inference",
)

# Lambda deployment setup
terraform_backend(name="s3_sandbox", source="sandbox.s3.tfbackend")
terraform_var_files(name="sandbox", sources=["sandbox.tfvars"])
terraform_module(name="test_lambda", dependencies=[":terraform_pants_lambda"])
terraform_deployment(name="terraform", root_module=":test_lambda", dependencies=[":terraform_pants_lambda"])

My intuition was that pants would recognise that the

:terraform_pants_lambda

target is a dependency of the terraform deployment and so it would build the package and make it available in the sandbox for deployment. Instead pants skips the dependency silently without warning. I figure either I have the wrong mental model about how pants works or this is a bug and my bet is on the former. My mental model was "dependencies of a target are made available in the sandbox" but that's clearly not always the case with some target types. Can someone point me at some docs or give me a nudge to help improve my understanding? I've been re-reading through the docs again and I haven't seen this concept raised,

Pants infers dependencies by scanning

import

statements in your source files. However when this is not available I would need to explicitly tell pants about certain target dependencies in the BUILD file that are generated with

pants tailor

as an example

python_test_utils(
    name="test_utils",
)

python_sources()

python_tests(
    name="tests0",
    dependencies=[
        "//:root#pytest-asyncio",
    ],
)

pytest-asyncio

It's a pytest plugin, not something your test files

import

directly. Tests use it via

@pytest.mark.asyncio

or the

asyncio_mode

setting, which pytest discovers through its plugin system at runtime not through imports. Hope this helps

Hi @acoustic-spring-13969 thanks for the reply, however I think you may have misunderstood where I'm getting confused. As you mentioned pants will try to infer dependencies, but in my example about there is a Terraform module that depends on an output file from a lambda packaging step. Pants can't infer that, so I've added the dependency explicitly to my Terraform module and deployment:

terraform_module(name="test_lambda", dependencies=[":terraform_pants_lambda"])
terraform_deployment(name="terraform", root_module=":test_lambda", dependencies=[":terraform_pants_lambda"])

However, even after explicitly listing the lambda target as a dependency, the output file from the lambda packaging step is not being materialised in the sandbox. That's the part I don't understand about pants dependencies and where I need some help with building an intuition for these things. Please do let me know if I've missed your point.

In Pants, the consuming target's rule implementation decides what it extracts from its dependencies*.* Adding something to

dependencies=

only puts it in the dependency graph but it will not automatically build and materialize output artifacts. Each backend's rule implementation decides what to extract. Terraform's rules only extract source files they never call

package()

on dependencies. This is arguably a gap in the terraform backend, not how all of Pants works. The Docker backend, by contrast, explicitly finds and builds packageable dependencies.

Yes, I think @acoustic-spring-13969 is correct here. Your mental model is the same as mine, and this feels like a lacuna in the Terraform backend - it should be gathering packages and it isn't.

Thanks both this is super interesting. As an aside, I would love to read more about this design decision:

the consuming target's rule implementation decides what it extracts from its dependencies

To me it makes a lot of sense: why materialise files that the consuming target doesn't know how to use? But it also feels to me like there's something unresolved from a user experience point of view here. Perhaps it's a documentation thing - knowing what are hard and fast pants behaviours vs target behaviours might be something useful to outline?

I agree that this should not be part of the learning process. I think this can be considered a bug in the tf backend, although there may have been a good reason for it. I think the general rule-of-thumb is "if any of my transitive deps are a package, build it and stick it in the downstream dependencies as a built file/dir"