Working on the dependencies issue for `experimental shell co

Question

Working on the dependencies issue for `experimental shell command` and realising that there s two purposes for dependencies here there s the runtime dependencies of your shell command these rarely need to be consumed by consumers of your command s `output`s since they re consumed and potentially transformed by the command into the outputs The dependencies that may make sense as being included in a transitive walk should only be the depednencies of the output itself and should therefore just be included as outputs I am leaning towards making it so that dependencies of an `experimental shell command` are not included in a transitive dependency resolution and that any dependencies of an `experimental shell command` that need to be consumed by subsequent steps need to be specified explicitly as an `output` Does that make sense < curved television 6568> < bitter ability 32190>

ancient-vegetable-10556 · Answer

the observation here is that you always need to pull the transitive dependencies to e g make a Python source with dependencies runnable but the outputs of shell commands almost always stand on their own

bitter-ability-32190 · Answer

almost always is dangerous to me here

bitter-ability-32190 · Answer

This touches deeply on the dependencies proposal IMO But my way of solving this would be having 2 deps fields

ancient-vegetable-10556 · Answer

I was leaning towards having two deps fields earlier too

fast-nail-55400 · Answer

or have the notion of scoped dependencies

ancient-vegetable-10556 · Answer

there s a proposal floating around for fancier dependencies already

fast-nail-55400 · Answer

like an earlier proposal of mine from way back to add a scope label parameter to dependency addresses

ancient-vegetable-10556 · Answer

I m just observing that for `experimental shell command` specifically the notion of transitive dependencies doesn t really make any sense

fast-nail-55400 · Answer

although yes there has been much thinking on that point subsequent to that way back proposal

ancient-vegetable-10556 · Answer

< bitter ability 32190> gt almost always is dangerous to me here Is there a specific concern that wouldn t be solved by explicitly including exportable dependencies as an output

ancient-vegetable-10556 · Answer

for the specific case of shell commands

ancient-vegetable-10556 · Answer

OK I have one and it leans towards having two dependencies fields one for runtime dependencies and one exportable dependencies

bitter-ability-32190 · Answer

I have one too but I need time to type it

fast-nail-55400 · Answer

is there any design consideration to be had here about how we would want to model dependency scopes for JVM target types

fast-nail-55400 · Answer

i e compile provided runtime test scopes

fast-nail-55400 · Answer

i e design this dependency solution to be consistent with what we would do for JVM

ancient-vegetable-10556 · Answer

tdyas probably and there s a design doc for me to point you at for the generic situation

ancient-vegetable-10556 · Answer

since this is an experimental feature I m happy to experiment with the design and veer towards the generic solution once we have it

ancient-vegetable-10556 · Answer

EXAMPLE You are generating HTTP API bindings for use in Python The bindings depend on `httpx` to make the HTTP request If you want to consume the bindings as Python sources the most Pants like solution would be to export `httpx` as a dependency of the outputs `httpx` however is not required by the bindings generator so it doesn t need to be requested when assembling the sandbox Likewise the bindings generator is not relevant to subsequent tasks

curved-television-6568 · Answer

the reasoning in here makes sense to me yea what I d like personally is if there where some way to attach meta data to the dependencies be it for scope or what not in a generic way that can be extended without too much effort i e for new kinds of meta information hopefully without becoming too bloated syntax wise

ancient-vegetable-10556 · Answer

< fast nail 55400> The relevant doc

bitter-ability-32190 · Answer

hes me

bitter-ability-32190 · Answer

My use case is using `esc` shortened to download an artifact that has other runtime deps I don t want those carried along with the downloaded thing because thats antithetical to Pants and transitive deps

ancient-vegetable-10556 · Answer

< bitter ability 32190> I think my forthcoming PR will do that It s adding an additional `runtime dependencies` field and a field to decide how to treat `dependencies`

bitter-ability-32190 · Answer

We have support `special cased dependencies`

bitter-ability-32190 · Answer

The proposal intends to treat them as less special stuck out tongue

ancient-vegetable-10556 · Answer

that makes this next bit of work a whole lot easier smile

bitter-ability-32190 · Answer

sweat smile Well my proposal won t be seeing any devtime soon disappointed

ancient-vegetable-10556 · Answer

Right that s why I figure diving in with a use case on an `experimental` feature is reasonably safe

ancient-vegetable-10556 · Answer

just wanted to land on a sane design before ploughing ahead

bitter-ability-32190 · Answer

The trick is most places dont ask for special deps So that means the runtime deps should be the special ones then the truly transitive deps are the normal ones

bitter-ability-32190 · Answer

You can change the name of the field to be explicit though

ancient-vegetable-10556 · Answer

yeah that s what I m doing

bitter-ability-32190 · Answer

booyah I look forward to a PR I ll shut up now slightly smiling face

ancient-vegetable-10556 · Answer

I think I may be tripping over myself a bit here but the results are great

bitter-ability-32190 · Answer

I literally have this in a plugin locally lol I just remembered

bitter-ability-32190 · Answer

It ll be nice to yeet that

ancient-vegetable-10556 · Answer

just got tests to do

ancient-vegetable-10556 · Answer

< bitter ability 32190> have at