#17729 Allow using a shell command as part of any goal (check, lint, fmt, fix, deploy, ...) Issue created by huonw Is your feature request related to a problem? Please describe. Shell is a universal glue, that allows augmenting pants adhoc with a lower barrier to entry (and, for simple tasks, lower maintenance burden) than writing a custom plugin. It'd be nifty to generalise the current codegen

experimental_shell_command

, run

experimental_run_shell_command

and recently test

experimental_test_shell_command

(#17640) to allow using a shell script for 'anything'. For example, in a polyglot repo, pants may support some of the languages/tools natively, but not others, but it'd still be nice to have basic commands like

./pants lint ::

and

./pants fmt ::

work across the whole system. Additionally, it would allow faster experiments and migration of an existing repo into pants, if glue scripts (and/or makefiles, or whatever) can be reused, and switch to having

./pants

to more. (We 'tripped' over both of these in our migration into pants, but I personally have liked pants enough to suffer through split tooling, and convince the rest of the team too, as well 😅 ) Describe the solution you'd like Something way to have a shell command hook into other goals. For instance:

# in a directory with CFN templates
experimental_shell_command(
   name="my-linter",
   command="cfn-lint",  # <https://github.com/aws-cloudformation/cfn-lint>, installed somehow
   tools=["..."],
   goal="check",
   dependencies=["./*.yaml"]
)

# in a docs/ directory
experimental_shell_command(
   name="my-formatter",
   command="mdformat",  # <https://github.com/executablebooks/mdformat>
   tools=["..."],
   goal="fmt",
   dependencies=["./**/*.md"]
)

• The

dependencies

would be the input files (for file-based commands). • The

outputs

field could be reused for

fmt

and

fix

to write back to the codebase. Some questions that seem relevant (although some can probably be deferred/considered potential future enhancements?): 1. How do these sort of shell commands get ordered with other `fmt`/`fix` subsystems (e.g. if

my-formatter

touches python files, does it run before or after black)? 2. Can a shell command delete files as part of

fmt

and

fix

? How does that work? 3. Maybe there can be opt-in or opt-out for invocations to run in parallel on subsets of the input? 1. How does the target distinguish between configuration file that needs to be in all invocations, and an input file should be in exactly one invocation? 2. Are outputs just naively merged (i.e. if one outputs

a.txt

and another outputs

b.txt

, the final output has both)? What happens if two invocations generate the same path (with different or identical content)? 4. Maybe lumping all goals into one command isn't quite right, since they may have very different options? 5. Is it easy/possible to express dependencies like

**/*.py

(to be able to run a custom linter on all Python files in descendent directories, without writing out the path to their individual targets)? 6. How do custom commands get installed? Some possibilities: • #17277 (or similar) for downloadable tools • a docker image environment • in-repo executables (#17405 may be relevant) Describe alternatives you've considered We are: • Actively using: shell scripts outside pants, with some sort of orchestrator for them (e.g. GitHub Actions YAML, internal documentation to copy-paste from for us; potentially Make or similar too) • Considering: custom plugins Additional context N/A pantsbuild/pants