I've been wondering about the following: Today if

python_library

X depends on a

python_distribution

Y then X effectively depends on all of the libraries Y depends on, transitively, as sources, and nothing special happens with Y itself. You could replace it with its direct deps with no change in behavior. But what the author of that dep probably intended is for X to depend on the wheel built from Y. An example of where this is useful is when you have a

python_distribution

that builds native code via a custom

setup.py

(this is now possible! See https://github.com/pantsbuild/pants/pull/12250). AFAICT today we handle this as a special case, only for running tests, via

runtime_package_dependencies

. But it seems like we should do this generically?

Implementation wise, it's much easier to do via a separate field. NB that it is a semantic error to include the sources of a runtime_package_dep, and you should only include the built artifact. We would have to add lots of special casing to our generic dependencies code to filter like that UX wise, I also think it is worth the clarity to users that we are going to do something special here: build an artifact

Does that make `python_distribution`s a blackhole for dep inference?

I had weakly proposed dependencies that spelled out products. Bazel had this but I think its now deprecated or removed. IE:

dependencies=['a/target:address@whl']

Where `@`'s would be registered by rules and dependencies on `@`'s would get the corresponding registered rule run against them with the product being a file(set).

my understanding of bazel is that you would affect the “type” of your dep on something via putting it in either the

deps

or

data

list. @average-vr-56795 would know better about whether the syntax still exists.

I don't think bazel has ever offered that kind of syntax exactly - I know Buck does and calls them flavours. Bazel has a couple of approaches to this - the main one is the separate attribute thing (e.g.

deps

vs

runtime_deps

); another is that you can name explicit files in some contexts (e.g. in a pkg_zip rule you can reference

:foo

or

foo_deploy.jar

and the latter will be a fat jar of transitive deps) but that only really applies when you're depending on a literal file copy, rather than wanting metadata like a classpath. A key difference between Bazel and Pants here is that in Bazel each rule pushes providers up the graph (so the dependency decides in what ways it can be consumed, and the depending rule picks one of them), whereas Pants pulls rule classes up the graph (the depender can decide to do interesting transforms itself)

Yes - exactly, _deploy.jar - that's what I'm referring to @average-vr-56795. @witty-crayon-22786 the

@

is a distraction - I intend none of the cross meaning with the old

@

- just a disambiguator since I lazily assumed

.

may already be allowed in address names.

(FWIW "flavours" got kind of terribly overloaded in Buck - they ended up becoming both "I want this particular kind of output" and also "I want to reconfigure this target" - you would use flavours both to differentiate "I want a .a" vs "I want a .so" but also "I want an x86_64" vs "I want an arm64" - and while those are similar concepts, the combinatorial explosion of them in one "address space" got very confusing (and also inefficient)

re: push vs pull in the rule graph… the target API ended up being shaped somewhat similarly to bazel providers and/or the “multiple different output files” of a rule: you request a particular

Field

type for a target, and if it is not declared literally on the target, it can be computed for the target

I quite like the idea of modelling a

python_distribution

as something which in some way pretends to be a

python_library

without any dependencies as far as things depending on it are concerned, but retains the dependency information for other contexts where that's needed... I'm not sure what exactly that would look like model-wise these days

but also, i think that

Fields

are still conceptually treated as inputs… even when they might be a

PythonSources

field generated from a

ProtobufSources

field (example from HEAD)

The same as depending on a

files

target containing the built artifact

Except you still want dep inference, right? Or is that a separate type that would get consumed a separate way?

@happy-kitchen-89482: in order for

python_library

consuming

python_distribution

to mean different things in different contexts, we either need new syntax or new attributes. we don’t have the

data

vs

deps

split, so we only have one list. if we had two attributes you’d put the

python_distribution

in your

data

list if you wanted loose files, or

deps

if the other thing.

I guess we have the dep inference issue today with the special case for tests

I think the other reason for not wanting to treat it completely opaquely is things like classpath conflict checking

for the motivating use-case (building native code in a custom

setup.py

) we don't have to worry about dep inference creating conflicts

So the wheels we're taking about are pure native code, and if they vendor any Python they do some kind of shading-like thing?

Or the user accepts that there may be a syspath ambiguity

I'm imagining e.g. some packages where there's a pure Python version, but if you're happy to pick up a native dep you can get a more optimised version, and I'm not sure how having two copies of that would go down. But I guess always picking the most locally built wheel probably works fine. But I guess it could be cleaner to be able to explicitly notice the conflict. I guess this ends up looking like the

provides

attribute on v1 JVM rules? Tagging the explicit metadata so you can handle the conflict smoothly