https://pantsbuild.org/ logo
#general
Title
# general
a

average-breakfast-91545

03/21/2023, 8:25 AM
Scenario: I'm building a lambda function that depends on aws-lambda-powertools and boto3. Accordingly, I need to add an override to my python_dependencies to pull in jmespath and pydantic, which are undeclared transitive deps. When I deploy my lambda function, I want to elide those deps because they'll ship as part of a lambda layer. My lambda func is a pex_binary:
Copy code
pex_binary (
    name="token_refresher",
    layout="zipapp",
    dependencies=[
        "myhandler.py",
        "!!3rdparty/python#boto3",
        "!!3rdparty/python#aws-lambda-powertools"
    ]
)
I'd have expected that jmespath and pydantic would be excluded, because they're transitive dependencies of two excluded deps, but by default they're pulled into the pex, so I have to exclude those as well. Is that intentional behaviour?
e

enough-analyst-54434

03/21/2023, 11:00 AM
I think it's expected behavior, but only if you know Pants is 3rd party dumb. It only tracks 1st party sources transitively and its knowledge ends at direct 3rd party dependencies. It offloads handling of those (3rd party resolution, locking, subsetting) to tools. The only sign you might see of this if not in the know, besides the sign you ran into here, is running
pants dependencies --transitive <target>
.
I think this is definitely buggy though. Pants really ought to be able to handle transitivity 1st class including in 3rd party deps. At the least, doc about what
!!
can and can't do should be improved.
b

bitter-ability-32190

03/21/2023, 12:08 PM
That's surprising to me. What does
./pants paths --from=path/to/pex:address --to=path/to/reqs:jmespath#resolve
say (you'll need to fill in your own values)
a

average-breakfast-91545

03/21/2023, 12:12 PM
Copy code
$ pants paths --paths-from=src/dz/recsys/anomaly_flagger:main --paths-to=3rdparty/python:jmespath
12:11:12.01 [ERROR] 1 Exception encountered:

Engine traceback:
  in `paths` goal

KeyError: Address(3rdparty/python#aws-lambda-powertools)
?
Copy code
❯ pants peek 3rdparty/python#aws-lambda-powertools
[
  {
    "address": "3rdparty/python#aws-lambda-powertools",
    "target_type": "python_requirement",
    "dependencies": [
      "3rdparty/python:jmespath",
      "3rdparty/python:pydantic",
      "3rdparty/python/pyproject.toml",
      "lockfiles/python-default.lock:python-default"
    ],
    "dependencies_raw": [
      ":pydantic",
      ":jmespath",
      "3rdparty/python/pyproject.toml",
      "lockfiles/python-default.lock:python-default"
    ],
    "description": null,
    "modules": null,
    "requirements": [
      "aws-lambda-powertools[parser]<3.0.0,>=2.9.0"
    ],
    "resolve": null,
    "tags": null,
    "type_stub_modules": null
  }
]
b

bitter-ability-32190

03/21/2023, 1:05 PM
Does your code depends on AWS lambda power tools? (Also file a bug if you can about the key error, you can use --print-traceback to see the calling code) (Also you don't have to use --path prefix after the
paths
goal, after a goal the prefix is implied)
a

average-breakfast-91545

03/21/2023, 1:06 PM
The code does, yes, but I want to exclude it from the package because it'll be present in a lambda layer.
b

bitter-ability-32190

03/21/2023, 1:08 PM
Oh yeah forgot that part of OP 🙃
a

average-breakfast-91545

03/21/2023, 1:08 PM
I'll see if I can repro the bug. The stacktrace isn't terribly informative
Copy code
13:07:19.35 [ERROR] 1 Exception encountered:

Engine traceback:
  in select
    ..
  in pants.backend.project_info.paths.paths
    `paths` goal

Traceback (most recent call last):
  File "/home/bob/.cache/nce/d036c571b74670fb7cccdb2ee973b51625c6419b29dd5e2e5390d6267b873904/bindings/venvs/2.15.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 593, in native_engine_generator_send
    res = func.send(arg)
  File "/home/bob/.cache/nce/d036c571b74670fb7cccdb2ee973b51625c6419b29dd5e2e5390d6267b873904/bindings/venvs/2.15.0/lib/python3.9/site-packages/pants/backend/project_info/paths.py", line 133, in paths
    for path in find_paths_breadth_first(adjacency_lists, root.address, destination.address):
  File "/home/bob/.cache/nce/d036c571b74670fb7cccdb2ee973b51625c6419b29dd5e2e5390d6267b873904/bindings/venvs/2.15.0/lib/python3.9/site-packages/pants/backend/project_info/paths.py", line 73, in find_paths_breadth_first
    for dep in adjacency_lists[target]:
KeyError: Address(3rdparty/python#aws-lambda-powertools)
b

bitter-ability-32190

03/21/2023, 1:09 PM
Ok yeah it's either getting the dep from somewhere else (but we can't see that because of a pants bug) Or the transitive exclude doesnt exclude its deps. I'm not sure which way of handling this would be more intuitive 🫤
Ahh I think that's enough info for the bug. I bet it isn't handling the exclude correctly 😅
a

average-breakfast-91545

03/21/2023, 1:10 PM
Okies, I'll open a bug 🙂
b

bitter-ability-32190

03/21/2023, 1:21 PM
Maybe we maybe we need another exclusion, lol
!!!
a

average-breakfast-91545

03/21/2023, 1:21 PM
I genuinely almost tried that, so it's at least intuitive
b

bitter-ability-32190

03/21/2023, 1:21 PM
Hahahaha excellent
Soon enough I think we're trending towards "tagged dependencies" so we won't be stuck with string mucking for modifiers
3 Views