https://pantsbuild.org/ logo
#general
Title
# general
f

freezing-lamp-12123

11/14/2022, 8:09 PM
is anyone using a workflow engine/language on top of pants for complicated jobs? — in the context of github action workflows, pants seems to be natural for running individual actions — but not for “orchestrating” workflows — am i missing something?
c

curved-television-6568

11/14/2022, 8:10 PM
I’ve been looking at tekton for this, but nothing in use yet: https://tekton.dev/
f

freezing-lamp-12123

11/14/2022, 8:13 PM
yeah — hope would be for something that also runs naturally on a local system without necessarily involving a kubernetes cluster … ph well, enough projects to choose from — just wanted to see if anyone was somehow clever with pants to do this
c

curved-television-6568

11/14/2022, 8:19 PM
intriguing idea.. to model a workflow engine on top of the pants engine.. 🤔 💡
f

freezing-lamp-12123

11/14/2022, 8:22 PM
the primary thing i’m missing is modeling steps with inputs/outputs — which could be “instances” — with specific parameters for each individual target/goal referenced in your “workflow” but you seem to already be inspired and cooking, so interested in your take for sure 🍿
i am considering giving dagster.io a spin — but it just feels like you’re taking something that should be fairly simple — and throwing a juggernaut at it — but, it might just work — maybe it just is complex? 😁
h

happy-kitchen-89482

11/14/2022, 11:53 PM
We are actually looking at similar stuff to this, e.g., on top of
experimental_shell_command
, where steps can be run (concurrently where possible) either on the local system, or in a docker container on the local system, or in a remote docker container
A lot of the recent Environments work is a step in this direction
@freezing-lamp-12123 Can you provide a few more details on your use-case?
f

freezing-lamp-12123

11/15/2022, 7:47 AM
yes, i can — this is all in the context of producing configuration for a big matrix of environments so we have some steps, to first generate “shared” configuration for a given environment, for a specific version — so we only want to generate that one time for each environment then we want to use this shared/generic environment configuration to pass as input to generate specific instances of a configuration of which there could be ~50 - 100 of for a given environment then we want to run validation on top of all the generated “artifacts” — so it’s a lot of passing inputs and getting outputs to individual targets
c

cool-yacht-37128

11/15/2022, 8:05 AM
I've been scripting stuff using using Python invoke. Also separately if it's suitable via
experimental_shell_command
h

happy-kitchen-89482

11/15/2022, 10:02 PM
Interesting, so I’m wondering about the combo of
experimental_shell_command
and parametrization for this?
What are the initial inputs and the final outputs of the whole workflow?
Or, I suppose what I’m asking is, what acts on all those configurations once they’re created and validated?
cc @ancient-vegetable-10556 for another potential use-case for
experimental_shell_command