proud-dentist-22844
07/23/2021, 4:43 PMfmt
, lint
, package
, repl
, run
, test
, and typecheck
(and friends: https://www.pantsbuild.org/docs/reference-all-goals) For instance there are a bunch of scripts here: https://github.com/pantsbuild/pants/tree/main/build-support/bin
And other projects will have their own set of ad-hoc tasks that need to be done (regen constraints, or build a venv for external tooling, or version bumping, or ...)
With a Makefile
, these tasks get added as additional custom make targets. We could use pants plugins to add more goals, but a pants-plugin feels like overkill for some of these tasks that are not meant to ever pass through remote caching & execution.
But I really want ./pants
to be the single point of entry for developer tasks.
So, what if there was a lighter weight way to extend pants? Maybe an invoke
pants-plugin (or call it a do
plugin?) that allowed for running tasks written for http://www.pyinvoke.org/ ? Essentially, a way to have a collection of plain (or nearly plain) python scripts that pants can run locally? Maybe the plugin would be able to dynamically create --help
for all of the available scripts so that it's a bit more structured than ./pants run
, but still extremely flexible.hundreds-father-404
07/23/2021, 4:48 PM./pants custom-script
(goal syntax)
or something like
./pants run path/to/custom_script.py
proud-dentist-22844
07/23/2021, 4:49 PM./pants custom-script
or ./pants do custom-script
(something really short)proud-dentist-22844
07/23/2021, 4:52 PM./pants invoke custom-script
but that starts to feel less accessible (only 4 more chars, yes, but then why not just do invoke custom-script
? I want a way to have ./pants
be the entry point even if it uses invoke or something else to enable the feature).proud-dentist-22844
07/23/2021, 5:04 PMfast-nail-55400
07/23/2021, 5:19 PMnpm run
?fast-nail-55400
07/23/2021, 5:20 PM[scripts]
section in pants.toml for each of these scriptscurved-television-6568
07/23/2021, 5:37 PMnpm run
is grunt
with it’s tasks, etc.. just for inspiration.. 😉
Edit: this doc: https://www.pantsbuild.org/docs/plugins-run-goalproud-dentist-22844
07/23/2021, 6:04 PM./pants run
require a path to the script to run? That doc looks like it just enables new kinds of binaries to run via ./pants run
, right? But you still have to know the path to the script to run it. Does that offer any benefit over running the script directly?hundreds-father-404
07/23/2021, 6:06 PMDoesn't ./pants run require a path to the script to run?Yes
hat doc looks like it just enables new kinds of binaries to run via ./pants run, right?Yeah, that's correct. That guide is meant as a guide for how you might do something like get
./pants run script.sh
to work
--
It sounds like what you're after is a way to add new custom goals like ./pants release
or ./pants changelog
, but in a more accessible way than the Plugin APIproud-dentist-22844
07/23/2021, 6:07 PMhundreds-father-404
07/23/2021, 6:10 PMproud-dentist-22844
07/23/2021, 6:10 PMshould this be similar in spirit toThat might be an interesting extension to?npm run
./pants run
where you can register some "shortcut targets" in pants.toml and then use ./pants run <shortcut target>
to run the script without having to pass in the absolute path or full target.proud-dentist-22844
07/23/2021, 6:11 PMhappy-kitchen-89482
07/24/2021, 1:02 AMgenrule
? https://docs.bazel.build/versions/main/be/general.htmlhappy-kitchen-89482
07/24/2021, 1:02 AMhappy-kitchen-89482
07/24/2021, 1:03 AMproud-dentist-22844
07/24/2021, 1:09 AMproud-dentist-22844
07/24/2021, 1:09 AMproud-dentist-22844
07/24/2021, 1:11 AMquaint-gold-40000
07/26/2021, 10:08 AMhappy-kitchen-89482
07/26/2021, 6:29 PMwitty-crayon-22786
07/26/2021, 6:31 PMwitty-crayon-22786
07/26/2021, 6:45 PMhappy-kitchen-89482
07/26/2021, 7:57 PMhappy-kitchen-89482
07/26/2021, 7:58 PMproud-dentist-22844
07/26/2021, 7:59 PMhappy-kitchen-89482
07/26/2021, 8:05 PMhappy-kitchen-89482
07/26/2021, 8:05 PMproud-dentist-22844
07/26/2021, 8:07 PMproud-dentist-22844
07/31/2021, 3:27 PMInteractivePexProcess
? I build an invoke
pex, and then I need to make an InteractiveProcess(..., run_in_workspace=True)
but InteractiveProcess.from_process()
doesn't have a run_in_workspace
arg.
https://github.com/pantsbuild/pants/blob/main/src/python/pants/engine/process.py#L328
Is there a way to run a pex interactively in the workspace?proud-dentist-22844
07/31/2021, 3:31 PMInteractiveProcess(
argv=invoke_pex.argv,
env=invoke_pex.env,
input_digest=invoke_pex.input_digest,
run_in_workspace=True,
forward_signals_to_process=True,
)
proud-dentist-22844
07/31/2021, 3:31 PMproud-dentist-22844
07/31/2021, 3:31 PMproud-dentist-22844
07/31/2021, 3:32 PMproud-dentist-22844
07/31/2021, 3:32 PMproud-dentist-22844
07/31/2021, 3:42 PMInterpreterConstraints
instead of PexInterpreterConstraints
.proud-dentist-22844
07/31/2021, 3:48 PMPexRequest
in the docs. main
instead of entry_point
and main
has to be a MainSpecification
.proud-dentist-22844
07/31/2021, 4:48 PMwitty-crayon-22786
07/31/2021, 8:55 PMfigure out how passing args to from commandline to InteractiveProcess worksthis is done via what is known as “passthrough” args on a subsystem: https://www.pantsbuild.org/docs/rules-api-subsystems#passthrough
figure out how to get pants to pull --help details from the subprocess to merge, UX-wise, the --help output.this will be challenging, because options setup can’t run processes… you’d have to do it statically
proud-dentist-22844
08/01/2021, 7:02 AMinvoke
in the pants plugin instead of using a pex so that the integration can feel more seamless.proud-dentist-22844
08/04/2021, 3:47 AMhundreds-father-404
08/05/2021, 10:34 PM./pants do --cmd1 --cmd2
?
I'm being a little lazy not looking more directly into invoke's source code: do you expect that code to be ./pants
?proud-dentist-22844
08/05/2021, 10:44 PMhundreds-father-404
08/05/2021, 10:48 PMpantsd_invalidation_globs
in pants.toml
under [GLOBAL]
The options system does not in general wire up to file watching, tracked by https://github.com/pantsbuild/pants/issues/10360
but otherwise, I think this works!hundreds-father-404
08/05/2021, 10:52 PMHmm. I might have to embed invoke in the pants plugin instead of using a pex so that the integration can feel more seamless.I don't totally follow that. Is the concern the performance to create the Pex the first time?
hundreds-father-404
08/05/2021, 11:55 PMdo
goal. It keeps ./pants help goals
tighter, allows you to see all the options w/ ./pants help do
proud-dentist-22844
08/06/2021, 2:28 AM./pants do task1 --task1-option
?fast-nail-55400
08/06/2021, 7:44 AMpassthru
(pass through) args where any arg after --
on the command line is a pass-through arg.fast-nail-55400
08/06/2021, 7:50 AM./pants do task1 -- --task1-option
fast-nail-55400
08/06/2021, 7:52 AM--task1-option
should be available via .passthru
on one of the options instancesfast-nail-55400
08/06/2021, 7:53 AMGoalSubsystem
subclass for your do
goal but this is where my knowledge is lakcinghundreds-father-404
08/06/2021, 1:56 PMproud-dentist-22844
08/06/2021, 3:28 PM--
or -
prefix, doesn't it? Is there another way to register a positional argument?happy-kitchen-89482
08/09/2021, 6:34 PMhappy-kitchen-89482
08/09/2021, 6:35 PM--
in there?happy-kitchen-89482
08/09/2021, 6:35 PM./pants do -- task1 --task1-option
proud-dentist-22844
08/09/2021, 9:12 PM--
and I'm working on dynamically registering all the options so that they show up in pants' help output.proud-dentist-22844
08/09/2021, 9:15 PM./pants lint black
or maybe use something other than space to say "this is a subgoal"? ./pants lint.black
hundreds-father-404
08/09/2021, 9:20 PMfast-nail-55400
08/09/2021, 9:24 PMproud-dentist-22844
08/09/2021, 9:27 PMproud-dentist-22844
08/09/2021, 9:31 PM./invoke
entry point (or ./do
) that uses pants to setup the tool, and then run it. Not having to create a virtualenv (or the makefile to orchestrate that) to get up and running is really nice.proud-dentist-22844
08/09/2021, 9:32 PMhundreds-father-404
08/09/2021, 9:33 PM./pants
bash script to detect if ./pants do
is used and sanitize it to Pants optionsproud-dentist-22844
08/09/2021, 9:33 PMproud-dentist-22844
08/09/2021, 9:34 PMpants
script)hundreds-father-404
08/09/2021, 9:35 PMproud-dentist-22844
08/10/2021, 12:32 AM[GLOBAL].plugins
, can I query pants (from the cli in bash) for the path to a virtualenv / pex that includes that requirement? Or the PYTHONPATH I should use to access it?hundreds-father-404
08/10/2021, 2:21 AMwitty-crayon-22786
08/10/2021, 3:07 AMproud-dentist-22844
08/10/2021, 3:10 AM./invoke
or ./inv
or (my favorite) ./do
)
https://github.com/st2sandbox/st2/blob/pants/invoke
And the pants-plugin with the magic wrapper script:
https://github.com/st2sandbox/st2/blob/pants/pants-plugins/do_invoke/goal.py#L144-L182 (rough, but functional)proud-dentist-22844
08/10/2021, 3:11 AMproud-dentist-22844
08/10/2021, 3:13 AMproud-dentist-22844
08/10/2021, 2:16 PMproud-dentist-22844
08/10/2021, 7:58 PM[GLOBAL].plugins
, so I believe I can rely on those plugins being available on sys.path
somewhere, right?witty-crayon-22786
08/10/2021, 8:28 PMOnce pants becomes a binary, it will still need some kind of external path to holdyea… any plugin loading mechanism will involve them being added to, so I believe I can rely on those plugins being available on[GLOBAL].plugins
somewhere, right?sys.path
sys.path
, so that should continue to work afaikhappy-kitchen-89482
08/12/2021, 2:18 PM./pants lint.flake8 --foo
it actually ran the entire lint
goal, for all linters, but allowed --foo
to refer to just the lint.flake8
scope. What @proud-dentist-22844 is proposing is different I think?proud-dentist-22844
08/12/2021, 2:34 PMlint.flake8
a thing?proud-dentist-22844
08/12/2021, 2:36 PMwitty-crayon-22786
08/12/2021, 3:47 PM./pants lint --only=flake8 ..
proud-dentist-22844
08/12/2021, 3:49 PMlint.flake8
would be a nice way to target “subgoals”, essentially saying that each linter is a subgoal of lint
. Then, I could use those subgoals elsewhere too.proud-dentist-22844
08/12/2021, 3:50 PM--only
is the next best experience imo.hundreds-father-404
08/12/2021, 3:55 PM./pants lint.flake8 lint.isort
? vs ./pants --lint-only=isort,flake8
probablyproud-dentist-22844
08/12/2021, 4:05 PM./pants lint fmt test
? If so, then lint.flake8 lint.isort fmt.black
could make sense.hundreds-father-404
08/12/2021, 4:06 PM./pants fmt lint test
!proud-dentist-22844
08/12/2021, 4:06 PMinvoke
for multiple goals + their options.proud-dentist-22844
08/12/2021, 4:06 PMproud-dentist-22844
08/12/2021, 4:07 PM./pants fmt <options for fmt> lint <options for lint> test <options for test>
hundreds-father-404
08/12/2021, 4:07 PMfmt
can overwrite files and invalidate the rest)proud-dentist-22844
08/12/2021, 4:11 PM<goal>.<subgoal>
would enable some cool shell shortcuts.
./pants lint.{flake8,isort,black} fmt.black
witty-crayon-22786
08/12/2021, 4:32 PMWith invoke, options are positional. Translating that to pants would be something like:they are in Pants in general as well… we used to have a page explaining this, but any options shown by./pants fmt <options for fmt> lint <options for lint> test <options for test>
./pants help $goal
can be used positionally with that goalwitty-crayon-22786
08/12/2021, 4:33 PMwitty-crayon-22786
08/12/2021, 4:34 PM--loop
option, which watches the filesystem and re-runs your goals. when i’m iterating on a test, i usually use:
./pants --loop test typecheck fmt lint $test
proud-dentist-22844
08/12/2021, 6:12 PM--help
output somehow. I've been looking at that and assumed I had to always be extremely verbose: --lint-*
proud-dentist-22844
08/12/2021, 6:19 PM--help
output more than I thought. LOLbusy-vase-39202
08/12/2021, 6:28 PMwitty-crayon-22786
08/12/2021, 6:40 PM./pants help global
… bare ./pants help
doesn’t show much at allwitty-crayon-22786
08/12/2021, 6:40 PMcurved-television-6568
10/10/2021, 7:21 AMexperimental_shell_command
(which is kind of like the Bazel genrule
Benjy mentioned) if used with ./pants export-codegen
could work pretty well as the entry point to such scripts.
Now to make it user friendly, I think that there could be a new “alias” subsystem, where you can configure cli arg replacements. I.e.
[alias.affected]
args = "--changed-since=origin/boss --changed-dependees=transitive"
Then you may use that as ./pants affected test
So with this, we could infer aliases based on shell command targets in, say a scripts directory.
/scripts/some-task.sh
#scripts/BUILD
experimental_shell_command(name="some-task", command="./some-task.sh", outputs=[…])
Then ./pants export-codegen scripts:some-task
could be aliased to ./pants do some-task
or even ./pants some-task
This has some cool side effects, like that you can depend on other targets for your tasks, etc. Also, if we add a output_path to the shell command, it should be allowed to replace/update/create files in the workspace, rather than under dist/
.
Also, the task may be run by remote execution, just as any other target.
In summary, we need support for aliases (which I think would be generally useful regardless of this idea) and some tweaks to the shell command.
Would be happy to work on this, if there’s agreement that it’s a cool idea :)witty-crayon-22786
10/11/2021, 5:46 PMwitty-crayon-22786
10/11/2021, 5:48 PMrun
support for experimental_shell_command
, or add a second target type (sharing a lot of the implementation), which is more parallel to the other *_binary
types, and explicitly about being run
in the foregroundcurved-television-6568
10/11/2021, 6:22 PMexperimental_shell_command
.curved-television-6568
10/12/2021, 11:43 AMcurved-television-6568
10/12/2021, 1:25 PMcurved-television-6568
10/12/2021, 1:42 PMproud-dentist-22844
10/13/2021, 6:15 PMproud-dentist-22844
10/13/2021, 6:17 PMproud-dentist-22844
10/13/2021, 6:19 PMexperimental_shell_command
+ aliases for invoke would look, but I would be interested in something along those lines for invoke.curved-television-6568
10/13/2021, 6:26 PM./invoke …
directly, rather than using ./pants invoke …
, right?curved-television-6568
10/13/2021, 6:30 PM./pants do <invoke-task> <task args>
proud-dentist-22844
10/13/2021, 11:14 PMproud-dentist-22844
10/13/2021, 11:15 PMproud-dentist-22844
10/13/2021, 11:16 PMproud-dentist-22844
10/13/2021, 11:17 PM./invoke
be the entry point for that, and it uses pants under-the-covers to manage things.fast-nail-55400
10/13/2021, 11:24 PMMaking pants aware of the invoke args proved quite problematic,were there any particular obstacles?
proud-dentist-22844
10/13/2021, 11:39 PM--
proud-dentist-22844
10/13/2021, 11:40 PM--
, or I had to go through a lot of hoops to grab the dynamic list of tasks available by importing invoke from within the plugin.proud-dentist-22844
10/13/2021, 11:40 PMproud-dentist-22844
10/13/2021, 11:43 PMcurved-television-6568
10/14/2021, 5:36 AM[cli.alias]
do = “run :invoke --“
So any args after do
will be pass through args to invoke, so as I suggested above, this could look like ./pants do some-task task-arg …
. So it only takes one additional level with the “do” arg to pants.proud-dentist-22844
10/14/2021, 2:07 PMcurved-television-6568
10/14/2021, 2:26 PM$ ./pants do hello
Hi from invoke task!
Traceback (most recent call last):
...
invoke.exceptions.ThreadException:
Saw 1 exceptions within threads (OSError):
Thread args: {'kwargs': {'echo': None,
'input_': <_io.TextIOWrapper name='<stdin>' mode='r' encoding='utf-8'>,
'output': <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>},
'target': <bound method Runner.handle_stdin of <invoke.runners.Local object at 0x10f2947c0>>}
...
return os.getpgrp() == os.tcgetpgrp(stream.fileno())
OSError: [Errno 25] Inappropriate ioctl for device
hundreds-father-404
10/14/2021, 2:33 PM--no-pantsd
. IPython fails with this same error when using Pantsd 😞 One of our most egregious broken windowscurved-television-6568
10/14/2021, 2:37 PM[cli.alias]
do = "--no-pantsd run :invoke --"
It works nicely.
$ ./pants do hello
Hi from invoke task!
proud-dentist-22844
10/14/2021, 4:38 PM