https://pantsbuild.org/ logo
a

ancient-vegetable-10556

12/07/2022, 5:52 PM
Working on the dependencies issue for
experimental_shell_command
, and realising that there’s two purposes for dependencies here — there’s the runtime dependencies of your shell command: these rarely need to be consumed by consumers of your command’s `output`s, since they’re consumed and (potentially) transformed by the command into the outputs. The dependencies that may make sense as being included in a transitive walk should only be the depednencies of the output itself, and should therefore just be included as outputs. I am leaning towards making it so that dependencies of an
experimental_shell_command
are not included in a transitive dependency resolution, and that any dependencies of an
experimental_shell_command
that need to be consumed by subsequent steps need to be specified explicitly as an
output
. Does that make sense? (@curved-television-6568, @bitter-ability-32190)
1
the observation here is that you always need to pull the transitive dependencies to e.g. make a Python source with dependencies runnable, but the outputs of shell commands almost always stand on their own
b

bitter-ability-32190

12/07/2022, 5:54 PM
almost always is dangerous to me here
This touches deeply on the dependencies proposal, IMO. But my way of solving this would be having 2 deps fields
a

ancient-vegetable-10556

12/07/2022, 5:55 PM
I was leaning towards having two deps fields earlier too
f

fast-nail-55400

12/07/2022, 5:55 PM
or have the notion of scoped dependencies
a

ancient-vegetable-10556

12/07/2022, 5:55 PM
there’s a proposal floating around for fancier dependencies already
f

fast-nail-55400

12/07/2022, 5:56 PM
like an earlier proposal of mine from way back to add a scope label/parameter to dependency addresses
a

ancient-vegetable-10556

12/07/2022, 5:56 PM
I’m just observing that for
experimental_shell_command
specifically, the notion of transitive dependencies doesn’t really make any sense
f

fast-nail-55400

12/07/2022, 5:56 PM
although yes there has been much thinking on that point subsequent to that way back proposal
a

ancient-vegetable-10556

12/07/2022, 5:58 PM
@bitter-ability-32190
almost always is dangerous to me here
Is there a specific concern that wouldn’t be solved by explicitly including exportable dependencies as an output?
(for the specific case of shell commands)
OK, I have one, and it leans towards having two dependencies fields: one for runtime dependencies and one exportable dependencies
b

bitter-ability-32190

12/07/2022, 6:04 PM
I have one too, but I need time to type it
f

fast-nail-55400

12/07/2022, 6:06 PM
is there any design consideration to be had here about how we would want to model dependency scopes for JVM target types?
(i.e., compile, provided, runtime, test scopes)
i.e., design this dependency solution to be consistent with what we would do for JVM
a

ancient-vegetable-10556

12/07/2022, 6:07 PM
tdyas, probably, and there’s a design doc for me to point you at for the generic situation
since this is an experimental feature, I’m happy to experiment with the design and veer towards the generic solution once we have it
EXAMPLE: You are generating HTTP API bindings for use in Python. The bindings depend on
httpx
to make the HTTP request. If you want to consume the bindings as Python sources, the most Pants-like solution would be to export
httpx
as a dependency of the outputs.
httpx
, however, is not required by the bindings generator, so it doesn’t need to be requested when assembling the sandbox. Likewise, the bindings generator is not relevant to subsequent tasks.
c

curved-television-6568

12/07/2022, 6:15 PM
the reasoning in here makes sense to me, yea. what I’d like personally is if there where some way to attach meta data to the dependencies, be it for “scope” or what not, in a generic way that can be extended without too much effort (i.e. for new kinds of meta information) hopefully without becoming too bloated syntax wise.
a

ancient-vegetable-10556

12/07/2022, 6:16 PM
b

bitter-ability-32190

12/07/2022, 6:45 PM
hes me
My use-case is using `esc`(shortened) to download an artifact that has other runtime deps. I don't want those carried along with the downloaded thing because thats antithetical to Pants and transitive deps
a

ancient-vegetable-10556

12/07/2022, 7:10 PM
@bitter-ability-32190 I think my forthcoming PR will do that. It’s adding an additional
runtime_dependencies
field, and a field to decide how to treat
dependencies
b

bitter-ability-32190

12/07/2022, 7:11 PM
We have support...
special_cased_dependencies
The proposal intends to treat them as.... less special 😛
a

ancient-vegetable-10556

12/07/2022, 7:12 PM
… that makes this next bit of work a whole lot easier 😄
b

bitter-ability-32190

12/07/2022, 7:14 PM
😅 Well my proposal won't be seeing any devtime soon 😞
a

ancient-vegetable-10556

12/07/2022, 7:14 PM
Right, that’s why I figure diving in with a use case on an
experimental
feature is reasonably safe
just wanted to land on a sane design before ploughing ahead
b

bitter-ability-32190

12/07/2022, 7:47 PM
The trick is most places dont ask for special deps. So that means the runtime deps should be the special ones, then the truly-transitive deps are the "normal" ones
You can change the name of the field to be explicit though
a

ancient-vegetable-10556

12/07/2022, 7:47 PM
yeah, that’s what I’m doing
b

bitter-ability-32190

12/07/2022, 7:47 PM
booyah. I look forward to a PR. I'll shut up now 🙂
a

ancient-vegetable-10556

12/07/2022, 8:05 PM
I think I may be tripping over myself a bit here, but the results are great.
b

bitter-ability-32190

12/07/2022, 8:07 PM
I literally have this in a plugin locally, lol I just remembered
It'll be nice to yeet that
🥨 1
a

ancient-vegetable-10556

12/07/2022, 8:07 PM
just got tests to do
@bitter-ability-32190 have at: https://github.com/pantsbuild/pants/pull/17743