https://pantsbuild.org/ logo
Join Slack
Powered by
Working on the dependencies issue for `experimenta...
# development
a
Working on the dependencies issue for 
experimental_shell_command
, and realising that there’s two purposes for dependencies here — there’s the runtime dependencies of your shell command: these rarely need to be consumed by consumers of your command’s `output`s, since they’re consumed and (potentially) transformed by the command into the outputs. The dependencies that may make sense as being included in a transitive walk should only be the depednencies of the output itself, and should therefore just be included as outputs. I am leaning towards making it so that dependencies of an 
experimental_shell_command
are not included in a transitive dependency resolution, and that any dependencies of an 
experimental_shell_command
that need to be consumed by subsequent steps need to be specified explicitly as an 
output
. Does that make sense? (@curved-television-6568, @bitter-ability-32190)
✅ 1
the observation here is that you always need to pull the transitive dependencies to e.g. make a Python source with dependencies runnable, but the outputs of shell commands almost always stand on their own
b
almost always is dangerous to me here
This touches deeply on the dependencies proposal, IMO. But my way of solving this would be having 2 deps fields
a
I was leaning towards having two deps fields earlier too
f
or have the notion of scoped dependencies
a
there’s a proposal floating around for fancier dependencies already
f
like an earlier proposal of mine from way back to add a scope label/parameter to dependency addresses
a
I’m just observing that for 
experimental_shell_command
specifically, the notion of transitive dependencies doesn’t really make any sense
f
although yes there has been much thinking on that point subsequent to that way back proposal
a
@bitter-ability-32190
almost always is dangerous to me here
Is there a specific concern that wouldn’t be solved by explicitly including exportable dependencies as an output?
(for the specific case of shell commands)
OK, I have one, and it leans towards having two dependencies fields: one for runtime dependencies and one exportable dependencies
b
I have one too, but I need time to type it
f
is there any design consideration to be had here about how we would want to model dependency scopes for JVM target types?
(i.e., compile, provided, runtime, test scopes)
i.e., design this dependency solution to be consistent with what we would do for JVM
a
tdyas, probably, and there’s a design doc for me to point you at for the generic situation
since this is an experimental feature, I’m happy to experiment with the design and veer towards the generic solution once we have it
EXAMPLE: You are generating HTTP API bindings for use in Python. The bindings depend on 
httpx
to make the HTTP request. If you want to consume the bindings as Python sources, the most Pants-like solution would be to export 
httpx
as a dependency of the outputs. 
httpx
, however, is not required by the bindings generator, so it doesn’t need to be requested when assembling the sandbox. Likewise, the bindings generator is not relevant to subsequent tasks.
c
the reasoning in here makes sense to me, yea. what I’d like personally is if there where some way to attach meta data to the dependencies, be it for “scope” or what not, in a generic way that can be extended without too much effort (i.e. for new kinds of meta information) hopefully without becoming too bloated syntax wise.
a
b
hes me
My use-case is using `esc`(shortened) to download an artifact that has other runtime deps. I don't want those carried along with the downloaded thing because thats antithetical to Pants and transitive deps
a
@bitter-ability-32190 I think my forthcoming PR will do that. It’s adding an additional 
runtime_dependencies
field, and a field to decide how to treat 
dependencies
b
We have support... 
special_cased_dependencies
The proposal intends to treat them as.... less special 😛
a
… that makes this next bit of work a whole lot easier 😄
b
😅 Well my proposal won't be seeing any devtime soon 😞
a
Right, that’s why I figure diving in with a use case on an 
experimental
feature is reasonably safe
just wanted to land on a sane design before ploughing ahead
b
The trick is most places dont ask for special deps. So that means the runtime deps should be the special ones, then the truly-transitive deps are the "normal" ones
You can change the name of the field to be explicit though
a
yeah, that’s what I’m doing
b
booyah. I look forward to a PR. I'll shut up now 🙂
a
I think I may be tripping over myself a bit here, but the results are great.
b
I literally have this in a plugin locally, lol I just remembered
It'll be nice to yeet that
🥨 1
a
just got tests to do
@bitter-ability-32190 have at: https://github.com/pantsbuild/pants/pull/17743
Open in Slack
PreviousNext
Powered by