Scenario: I'm building a lambda function that depends on aws-lambda-powertools and boto3. Accordingly, I need to add an override to my python_dependencies to pull in jmespath and pydantic, which are undeclared transitive deps. When I deploy my lambda function, I want to elide those deps because they'll ship as part of a lambda layer. My lambda func is a pex_binary:

pex_binary (
    name="token_refresher",
    layout="zipapp",
    dependencies=[
        "myhandler.py",
        "!!3rdparty/python#boto3",
        "!!3rdparty/python#aws-lambda-powertools"
    ]
)

I'd have expected that jmespath and pydantic would be excluded, because they're transitive dependencies of two excluded deps, but by default they're pulled into the pex, so I have to exclude those as well. Is that intentional behaviour?

I think it's expected behavior, but only if you know Pants is 3rd party dumb. It only tracks 1st party sources transitively and its knowledge ends at direct 3rd party dependencies. It offloads handling of those (3rd party resolution, locking, subsetting) to tools. The only sign you might see of this if not in the know, besides the sign you ran into here, is running

pants dependencies --transitive <target>

.

That's surprising to me. What does

./pants paths --from=path/to/pex:address --to=path/to/reqs:jmespath#resolve

say (you'll need to fill in your own values)

$ pants paths --paths-from=src/dz/recsys/anomaly_flagger:main --paths-to=3rdparty/python:jmespath
12:11:12.01 [ERROR] 1 Exception encountered:

Engine traceback:
  in `paths` goal

KeyError: Address(3rdparty/python#aws-lambda-powertools)

?

❯ pants peek 3rdparty/python#aws-lambda-powertools
[
  {
    "address": "3rdparty/python#aws-lambda-powertools",
    "target_type": "python_requirement",
    "dependencies": [
      "3rdparty/python:jmespath",
      "3rdparty/python:pydantic",
      "3rdparty/python/pyproject.toml",
      "lockfiles/python-default.lock:python-default"
    ],
    "dependencies_raw": [
      ":pydantic",
      ":jmespath",
      "3rdparty/python/pyproject.toml",
      "lockfiles/python-default.lock:python-default"
    ],
    "description": null,
    "modules": null,
    "requirements": [
      "aws-lambda-powertools[parser]<3.0.0,>=2.9.0"
    ],
    "resolve": null,
    "tags": null,
    "type_stub_modules": null
  }
]

Does your code depends on AWS lambda power tools? (Also file a bug if you can about the key error, you can use --print-traceback to see the calling code) (Also you don't have to use --path prefix after the

paths

goal, after a goal the prefix is implied)

The code does, yes, but I want to exclude it from the package because it'll be present in a lambda layer.

Oh yeah forgot that part of OP 🙃

I'll see if I can repro the bug. The stacktrace isn't terribly informative

13:07:19.35 [ERROR] 1 Exception encountered:

Engine traceback:
  in select
    ..
  in pants.backend.project_info.paths.paths
    `paths` goal

Traceback (most recent call last):
  File "/home/bob/.cache/nce/d036c571b74670fb7cccdb2ee973b51625c6419b29dd5e2e5390d6267b873904/bindings/venvs/2.15.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 593, in native_engine_generator_send
    res = func.send(arg)
  File "/home/bob/.cache/nce/d036c571b74670fb7cccdb2ee973b51625c6419b29dd5e2e5390d6267b873904/bindings/venvs/2.15.0/lib/python3.9/site-packages/pants/backend/project_info/paths.py", line 133, in paths
    for path in find_paths_breadth_first(adjacency_lists, root.address, destination.address):
  File "/home/bob/.cache/nce/d036c571b74670fb7cccdb2ee973b51625c6419b29dd5e2e5390d6267b873904/bindings/venvs/2.15.0/lib/python3.9/site-packages/pants/backend/project_info/paths.py", line 73, in find_paths_breadth_first
    for dep in adjacency_lists[target]:
KeyError: Address(3rdparty/python#aws-lambda-powertools)

Ok yeah it's either getting the dep from somewhere else (but we can't see that because of a pants bug) Or the transitive exclude doesnt exclude its deps. I'm not sure which way of handling this would be more intuitive 🫤

Okies, I'll open a bug 🙂

Maybe we maybe we need another exclusion, lol

!!!

I genuinely almost tried that, so it's at least intuitive

Hahahaha excellent