Filtering dependencies of package targets seems to come up regularly (

python_distribution

,

pex_binary

,

debian_package

, and now the nfpm package types I'm working on). In many cases, these packages depend on another package, we want to include the package/archive itself, not the sources packaged in the target or owned by the target. For

python_distribution

there's quite a bit of logic to determine which distribution "owns" a given python source file. Some package targets (like

debian_package

) use a SpecialCasedDependency field to exclude their deps from normal dep calculation. So, what if we allow passing a predicate (lambda function) when requesting transitive deps that says whether or not to follow the dependencies of a given target. Maybe something like this (probably with the predicate on

TransitiveTargetsRequest

instead of on the

_DependencyMappingRequest

)? https://github.com/cognifloyd/pants/commit/b4044e76685041cdac322a3e96f4858b4634f045 (For what it's worth, mypy is happy with this stub commit.) Do any other request types use a lambda like this? Is that going to mess up graph calculation some how? I tried to minimize the effect on the current predicate-less code path, but I'm not sure about the performance implications of messing with the dep graph here. Should I keep going with this approach?

This is a relevant issue: https://github.com/pantsbuild/pants/issues/12794

Yeah. That issue is much broader in "scope" (couldn't resist the pun).

So Instead of doing this for a specific scenario, I think we should just bite the bullet on the "tagged dependencies" idea.

@happy-kitchen-89482 doing this for all package types would mean importing

PackageFieldSet

into the engine internals which is a little gross. :/ Moving it is an option as well.

Tagged dependencies gives Pants more a clue to how a dependency is meant to be consumed (and therefore should be provided by Pants). In this case, we don't know whether the dev wants the target as a loose set of files (like say, I wanted to lint the files that go into a

pex_binary

) or as the packaged thing (I intend to upload this

artifact

using my super special script). Tagging allows that vital information to be known to Pants, therefore we make less assumptions (exactly 0 assumptions 😛)

I'm suggesting that we would add some way of knowing this

The whole point of our lengthy discussions earlier was that targets shouldn't/can't assume how they will be consumed. We shift that knowledge onto the consumer

imo, step one is a consuming predicate/API that allows for controlling which dependencies are traversed. step two is adding more information for that API to consume.

That............ damn. I've been Stu'd. Yeah that makes complete sense.

Something like this predicate? https://github.com/cognifloyd/pants/commit/b4044e76685041cdac322a3e96f4858b4634f045

@proud-dentist-22844: right. and sorry for not reading the whole thread when i got the ping =x

Whereas tagging dependencies would be an edge filter?

a predicate API that allowed for considering individual edges would be an edge filter, yea. it doesn’t require tagging: for example, maybe you want to include from one

Dependencies

field subtype or another

Is passing in a lambda on the request a kosher way for the consumer to pass its filter? (Whether the filter is node or edge)

I'm not suggesting that target's should know how they are consumed, I'm suggesting that in this case the traversal of an edge is NOT dependent on any property of that edge.

One interesting thing: With

python_distribution

traversing all the other

python_distribution

targets, if a python source under dist A imports python source under dist B, and dist B explicitly depends on dist A, then dist A will package that source even if none of the entry points depend on it. Once we stop traversing the graph through package deps, we may have to be much more explicit about what each dist includes (esp for library dists that don't have entry points).

sorry for the long delay. i added a comment that sets a somewhat arbitrary benchmark that i think would be good to hit with this filtering API before landing it: https://github.com/pantsbuild/pants/pull/19155#discussion_r1213566914 obviously this API doesn’t have to be perfect before landing. but IMO, ensuring that this API is sufficient to remove some existing complexity would be good. thanks for looking at it!

My review is still; pending 🤦‍♂️

I'm studying

SpecialCasedDependencies

, to figure out what use-cases it covers... 🤯 Handling

archive

and

debian_package

look fairly straight forward, and then it gets fuzzier but still doable (via tagged dependencies?) for

python_test(s)

,

shunit2_test(s)

,

pex_binar{y,ies}

,

python_{google_cloud,aws_lambda}_function

,

python_aws_lambda_layer

,

adhoc_tool

,

shell_command

,

run_shell_command

,

system_binary

, and

helm_deployment

. And then

wrap_source_as_*

and

relocated_files

need something a bit different, but probably still doable. But then there's

jvm_war

. Geez. The complexities around handling java are so painful.

It's a... War zone 😂

LOL 🤣

i think the easiest way to describe

SpecialCasedDependencies

is: “we want to support multiple subclasses of

Dependencies

on one Target”