https://pantsbuild.org/ logo
Join Slack
Powered by
Another feature idea (possibly in a plugin): There...
# general
p
Another feature idea (possibly in a plugin): There are a few developer tasks that don't fit within the pants goals: 
fmt
, 
lint
, 
package
, 
repl
, 
run
, 
test
, and 
typecheck
(and friends: https://www.pantsbuild.org/docs/reference-all-goals) For instance there are a bunch of scripts here: https://github.com/pantsbuild/pants/tree/main/build-support/bin And other projects will have their own set of ad-hoc tasks that need to be done (regen constraints, or build a venv for external tooling, or version bumping, or ...) With a 
Makefile
, these tasks get added as additional custom make targets. We could use pants plugins to add more goals, but a pants-plugin feels like overkill for some of these tasks that are not meant to ever pass through remote caching & execution. But I really want 
./pants
to be the single point of entry for developer tasks. So, what if there was a lighter weight way to extend pants? Maybe an 
invoke
pants-plugin (or call it a 
do
plugin?) that allowed for running tasks written for http://www.pyinvoke.org/ ? Essentially, a way to have a collection of plain (or nearly plain) python scripts that pants can run locally? Maybe the plugin would be able to dynamically create 
--help
for all of the available scripts so that it's a bit more structured than 
./pants run
, but still extremely flexible.
✔️ 2
h
Interesting idea! What would be your ideal UX for how you invoke those custom scripts? 
./pants custom-script
(goal syntax) or something like 
./pants run path/to/custom_script.py
p
./pants custom-script
or 
./pants do custom-script
(something really short)
👍 1
Maybe 
./pants invoke custom-script
but that starts to feel less accessible (only 4 more chars, yes, but then why not just do 
invoke custom-script
? I want a way to have 
./pants
be the entry point even if it uses invoke or something else to enable the feature).
Context for my thoughts: My work moving StackStorm to pants overlaps with this: https://github.com/StackStorm/st2/pull/4782 (replace Makefile with invoke). But not all of these tasks fit neatly into one of the pants goals. So, I was trying to think of a way to reuse parts of this with pants. Thus the feature idea.
f
should this be similar in spirit to 
npm run
?
👍 1
🤔 1
i.e., just have a 
[scripts]
section in pants.toml for each of these scripts
c
there’s an example in the pants docs about writing a “binary run plugin” that executes .. stuff. Not entirely sure of its context, though (what it runs), just came to think of it. Also, along with the parallell to 
npm run
is 
grunt
with it’s tasks, etc.. just for inspiration.. 😉 Edit: this doc: https://www.pantsbuild.org/docs/plugins-run-goal
p
Doesn't 
./pants run
require a path to the script to run? That doc looks like it just enables new kinds of binaries to run via 
./pants run
, right? But you still have to know the path to the script to run it. Does that offer any benefit over running the script directly?
h
Doesn't ./pants run require a path to the script to run?
Yes
hat doc looks like it just enables new kinds of binaries to run via ./pants run, right?
Yeah, that's correct. That guide is meant as a guide for how you might do something like get 
./pants run script.sh
to work -- It sounds like what you're after is a way to add new custom goals like 
./pants release
or 
./pants changelog
, but in a more accessible way than the Plugin API
p
yeah. And that more accessible way doesn't need access to the rules or targets (or other pants) APIs.
h
My first intuition is: make the rules API simpler such that it is accessible, including: - fix rule graph errors - better docs and examples Prefer to improve our main API rather than bifurcating into more APIs. But it may be the case that it's still Too Much and we need a more limited API, which we should be open to imo
p
should this be similar in spirit to 
npm run
?
That might be an interesting extension to 
./pants run
where you can register some "shortcut targets" in pants.toml and then use 
./pants run <shortcut target>
to run the script without having to pass in the absolute path or full target.
The problem with using the rules API is that I'm supposed to think of it as a DSL, not raw python.
✔️ 1
h
Basically a generic goal to run scripts?
But then you have to take extra care around side effects and such?
p
It would be explicitly side-effecty
and therefore could not run via remote execution
So, yeah - it's sort of like genrule, but that sounds like a rule that could get triggered in the normal course of the other goals. I'm thinking more about things that would only run when explicitly requested.
q
I've found https://pydoit.org/ to be great for describing and running generic tasks.
👀 1
h
@witty-crayon-22786 Can we somehow mark processes as not-remotable today?
w
Only via InteractiveProcess, which also forces it to run in the foreground, and is only legal in a goal_rule
But that's exactly where you'd want "things that only run when explicitly requested"
h
And "things that can side-effect"
So @proud-dentist-22844 the idea is that no rules depend on the output of running these processes? Running them is the end goal itself?
p
yup
h
Then this should be pretty straightforward
Want to take a swipe at it? 🙂
p
Sure. Next time I'm hacking on st2+pants, I'll work on this 🙂
Is there something like 
InteractivePexProcess
? I build an 
invoke
pex, and then I need to make an 
InteractiveProcess(..., run_in_workspace=True)
but 
InteractiveProcess.from_process()
doesn't have a 
run_in_workspace
arg. https://github.com/pantsbuild/pants/blob/main/src/python/pants/engine/process.py#L328 Is there a way to run a pex interactively in the workspace?
Copy code
InteractiveProcess(
            argv=invoke_pex.argv,
            env=invoke_pex.env,
            input_digest=invoke_pex.input_digest,
            run_in_workspace=True,
            forward_signals_to_process=True,
        )
I think that's it
Except that you can't use input_digest and run_in_workspace together.
hmm
https://www.pantsbuild.org/docs/rules-api-installing-tools#pex-install-binaries-through-pip has an issue: it should use 
InterpreterConstraints
instead of 
PexInterpreterConstraints
.
Looks like there are a couple other thing that are out-of-date with that 
PexRequest
in the docs. 
main
instead of 
entry_point
and 
main
has to be a 
MainSpecification
.
Here's a skeleton for running invoke in the workspace: https://github.com/st2sandbox/st2/blob/pants/pants-plugins/do_invoke/goal.py Next steps: • figure out how passing args to from commandline to InteractiveProcess works • figure out how to get pants to pull --help details from the subprocess to merge, UX-wise, the --help output.
w
figure out how passing args to from commandline to InteractiveProcess works
this is done via what is known as “passthrough” args on a subsystem: https://www.pantsbuild.org/docs/rules-api-subsystems#passthrough
figure out how to get pants to pull --help details from the subprocess to merge, UX-wise, the --help output.
this will be challenging, because options setup can’t run processes… you’d have to do it statically
p
Hmm. I might have to embed 
invoke
in the pants plugin instead of using a pex so that the integration can feel more seamless.
What do you think of extracting the available options like this? https://github.com/st2sandbox/st2/blob/pants/pants-plugins/do_invoke/goal.py#L50-L102
h
Huh, interesting. Very creative! Are you envisioning this will be something like 
./pants do --cmd1 --cmd2
? I'm being a little lazy not looking more directly into invoke's source code: do you expect that code to be stable deterministic? To be sure, it doesn't depend on the argv you used to run 
./pants
?
p
I expect it to depend only on what tasks are available (invoke looks at the file system). It could depend on argv, but I bypassed all that stuff because I want something more deterministic. I guess the presence of tasks is similar to registering pants backends to enable additional commands.
👍 1
h
Cool. That's good. I suspect this code will not properly handle file watching, meaning if you're using pantsd and change the invoke files, it won't be picked up If that is indeed a problem, an awkward workaround is to add those files to 
pantsd_invalidation_globs
in 
pants.toml
under 
[GLOBAL]
The options system does not in general wire up to file watching, tracked by https://github.com/pantsbuild/pants/issues/10360 but otherwise, I think this works!
👀 1
Hmm. I might have to embed invoke in the pants plugin instead of using a pex so that the integration can feel more seamless.
I don't totally follow that. Is the concern the performance to create the Pex the first time?
Btw, I like the modeling of this being options on the 
do
goal. It keeps 
./pants help goals
tighter, allows you to see all the options w/ 
./pants help do
p
Is it possible to have non option arguments after the goal? like 
./pants do task1 --task1-option
?
f
Yes, there is a notion of 
passthru
(pass through) args where any arg after 
--
on the command line is a pass-through arg.
👍 1
so you would do: 
./pants do task1 -- --task1-option
then 
--task1-option
should be available via 
.passthru
on one of the options instances
I’d like to say on the 
GoalSubsystem
subclass for your 
do
goal but this is where my knowledge is lakcing
h
Look at black/subsystem.py as an example of pass through args
p
No passthrough args doesn't do what I'm looking for - I want to add a list of "sub" goals. But registering an option requires 
--
or 
-
prefix, doesn't it? Is there another way to register a positional argument?
h
There isn't atm - all cmd-line flags must be -x or --long-name
Why wouldn't passthru work? Is it that you don't like the floating 
--
in there?
So it would be 
./pants do -- task1 --task1-option
p
No, I don't like the floating 
--
and I'm working on dynamically registering all the options so that they show up in pants' help output.
Option registration should be working fine. A way to register sub-goals would be nice though. Ooh. Thinking about sub-goals, it would also be nice to target particular linters: 
./pants lint black
or maybe use something other than space to say "this is a subgoal"? 
./pants lint.black
h
Pants v1 used to have something similar to that, which is what Benjy just finished removing hehe It's plausible we'd want to add back but in a re-envisioned way
f
is there any reason to not just defer to the sub-tool for option parsing of tool-specific options?
p
To use pants as a seamless single-point-of-entry for dev tasks.
Maybe I'll have to add an extra 
./invoke
entry point (or 
./do
) that uses pants to setup the tool, and then run it. Not having to create a virtualenv (or the makefile to orchestrate that) to get up and running is really nice.
👍 1
Then there's more than one entry point, but the tool designs seem to conflict, so merging them is not working out.
h
This could be a bad idea, not sure I recommend it. Another option is to update 
./pants
bash script to detect if 
./pants do
is used and sanitize it to Pants options
p
hmm. Then updates for that script would become more difficult.
➕ 1
(lol - maybe one of the adhoc tasks could be to update the 
pants
script)
h
Yeah, and we do want to move to distributing Pants as a binary rather than PyPI (altho possibly the bash script would still exist, unclear)
p
GIven a 3rd party dep in pants.toml > 
[GLOBAL].plugins
, can I query pants (from the cli in bash) for the path to a virtualenv / pex that includes that requirement? Or the PYTHONPATH I should use to access it?
h
Hmm @witty-crayon-22786 do you know by chance?
w
hm, no… that’s a well kept secret / implementation detail, heh. plugins are resolved using PEX, but then referenced directly out of the PEX cache.
p
well, here's my invoke plugin v2, using a wrapper script: The wrapper script (accessible as 
./invoke
or 
./inv
or (my favorite) 
./do
) https://github.com/st2sandbox/st2/blob/pants/invoke And the pants-plugin with the magic wrapper script: https://github.com/st2sandbox/st2/blob/pants/pants-plugins/do_invoke/goal.py#L144-L182 (rough, but functional)
(I'm probably relying on a bunch of implementation details I shouldn't... lol)
I'm open to suggestions and yelling (things like "What on earth are you thinking!?" might be appropriate 😉 )
😆 1
@witty-crayon-22786 How safe/wise is creating this script: https://github.com/st2sandbox/st2/blob/pants/pants-plugins/do_invoke/goal.py#L75-L77 Which relies on the plugin here: https://github.com/st2sandbox/st2/blob/pants/pants.toml#L37 and gets called here: https://github.com/st2sandbox/st2/blob/pants/invoke#L16-L24 Am I shooting myself in the foot by doing this? Once pants becomes a binary, it will still need some kind of external path to hold 
[GLOBAL].plugins
, so I believe I can rely on those plugins being available on 
sys.path
somewhere, right?
w
Once pants becomes a binary, it will still need some kind of external path to hold 
[GLOBAL].plugins
 , so I believe I can rely on those plugins being available on 
sys.path
 somewhere, right?
yea… any plugin loading mechanism will involve them being added to 
sys.path
, so that should continue to work afaik
h
For the record, what I removed wasn't "subgoals", which we never had in v1, but just subscoping of options. So when you ran 
./pants lint.flake8 --foo
it actually ran the entire 
lint
goal, for all linters, but allowed 
--foo
to refer to just the 
lint.flake8
scope. What @proud-dentist-22844 is proposing is different I think?
👍 1
p
Wait is 
lint.flake8
a thing?
Could that be repurposed for subgoals? To run only the flake8 linter?
w
there was talk of having a way to do that… basically 
./pants lint --only=flake8 ..
p
yup. 
lint.flake8
would be a nice way to target “subgoals”, essentially saying that each linter is a subgoal of 
lint
. Then, I could use those subgoals elsewhere too.
If we didn’t do that, then 
--only
is the next best experience imo.
h
What would multiple linters look like? 
./pants lint.flake8 lint.isort
? vs 
./pants --lint-only=isort,flake8
probably
p
Is it possible to do multiple goals right now? 
./pants lint fmt test
? If so, then 
lint.flake8 lint.isort fmt.black
could make sense.
h
Yeah it's possible to do 
./pants fmt lint test
!
p
And if multiple goals isn’t possible at this point, maybe checkout the UX from 
invoke
for multiple goals + their options.
oh, it is? cool
With invoke, options are positional. Translating that to pants would be something like: 
./pants fmt <options for fmt> lint <options for lint> test <options for test>
h
(We have a TODO to allow those rules to run concurrently, which is an interesting problem because you have to keep track of things like how 
fmt
can overwrite files and invalidate the rest)
p
The 
<goal>.<subgoal>
would enable some cool shell shortcuts. 
./pants lint.{flake8,isort,black} fmt.black
w
With invoke, options are positional. Translating that to pants would be something like: 
./pants fmt <options for fmt> lint <options for lint> test <options for test>
they are in Pants in general as well… we used to have a page explaining this, but any options shown by 
./pants help $goal
can be used positionally with that goal
(the page still exists for v1, and ironically, it’s called: “Invoking Pants”: https://v1.pantsbuild.org/invoking.html)
we also need to document the 
--loop
option, which watches the filesystem and re-runs your goals. when i’m iterating on a test, i usually use:
Copy code
./pants --loop test typecheck fmt lint $test
p
Hmm. It also needs to be documented in the 
--help
output somehow. I've been looking at that and assumed I had to always be extremely verbose: 
--lint-*
@busy-vase-39202 I guess I really do rely on the 
--help
output more than I thought. LOL
b
Haha.
w
it’s in 
./pants help global
… bare 
./pants help
doesn’t show much at all
(…although it says it’s “alpha” which is no longer the case. will fix.)
c
@proud-dentist-22844 Have you made any progress on the invoke stuff? I’ve been piecing together a few ideas in my head, and just realized it could be an interesting approach to for this topic on how to run “ad-hoc” scripts. Given the new 
experimental_shell_command
(which is kind of like the Bazel 
genrule
Benjy mentioned) if used with 
./pants export-codegen
could work pretty well as the entry point to such scripts. Now to make it user friendly, I think that there could be a new “alias” subsystem, where you can configure cli arg replacements. I.e.
Copy code
[alias.affected]
args = "--changed-since=origin/boss --changed-dependees=transitive"
Then you may use that as 
./pants affected test
So with this, we could infer aliases based on shell command targets in, say a scripts directory.
Copy code
/scripts/some-task.sh

#scripts/BUILD
experimental_shell_command(name="some-task", command="./some-task.sh", outputs=[…])
Then 
./pants export-codegen scripts:some-task
could be aliased to 
./pants do some-task
or even 
./pants some-task
This has some cool side effects, like that you can depend on other targets for your tasks, etc. Also, if we add a output_path to the shell command, it should be allowed to replace/update/create files in the workspace, rather than under 
dist/
. Also, the task may be run by remote execution, just as any other target. In summary, we need support for aliases (which I think would be generally useful regardless of this idea) and some tweaks to the shell command. Would be happy to work on this, if there’s agreement that it’s a cool idea :)
👀 1
w
the aliases concept is a neat one… there was a previous proposal in that general area (although for fairly different reasons!), but this is even shorter.
👍 1
regarding using `export-codegen`here though: it seems like maybe we might want to add 
run
support for 
experimental_shell_command
, or add a second target type (sharing a lot of the implementation), which is more parallel to the other 
*_binary
types, and explicitly about being 
run
in the foreground
💯 1
c
Yeah, I like the idea of running a 
experimental_shell_command
.
And for shell command run support: https://github.com/pantsbuild/pants/pull/13229
There’s no alias inference based on any experimental_shell_command targets yet.. not really sure how that would work well (and effectively). Feeling that it might be good enough, and clearer, to require any scripts to be added as alias in the configuration explicitly.
p
OK. I took a look. Yes, I got my invoke goal working. https://github.com/st2sandbox/st2/tree/pants/pants-plugins/do_invoke It uses a bit of a hack to install invoke in the pants venv, and then run it from there to skip pex-ing anything and get it to run as fast as possible. As such, it does not work with remote execution, as I went with the assumption that invoke tasks are local-only / dev-only tasks.
👀 1
aliases would be really cool though 🙂
I’m not sure how codegen + 
experimental_shell_command
+ aliases for invoke would look, but I would be interested in something along those lines for invoke.
c
Interesting, so you use pants to create the invoke wrapper script, and then run 
./invoke …
directly, rather than using 
./pants invoke …
, right?
I think it would be pretty easy to wrap up invoke in a pex as a dependency to a shell command, and run that with pants, and aliases to make any invoke task runnable as 
./pants do <invoke-task> <task args>
p
No. Pants invokes the wrapper script as well. https://github.com/st2sandbox/st2/blob/pants/pants-plugins/do_invoke/goal.py#L97 but it doesn't need to recreate the wrapper script because that is cached.
ah. No nevermind. You're right. It's been too long since I looked at this.
Making pants aware of the invoke args proved quite problematic, so I opted to have 
./invoke
be the entry point for that, and it uses pants under-the-covers to manage things.
f
Making pants aware of the invoke args proved quite problematic,
were there any particular obstacles?
p
--
I either had to sacrifice exposing things in help, and use 
--
, or I had to go through a lot of hoops to grab the dynamic list of tasks available by importing invoke from within the plugin.
Also, invoke has the concept of targets and sub targets, so I would have needed some way to create sub-goals, but pants doesn't have that.
I guess, to use the alias feature, you'd end up registering each invoke target as an alias to reduce the number of levels you have to explicitly add on the command line to get to the function you want.
👍 1
c
I’m not yet familiar with python-invoke, but how about
Copy code
[cli.alias]
do = “run :invoke --“
So any args after 
do
will be pass through args to invoke, so as I suggested above, this could look like 
./pants do some-task task-arg …
. So it only takes one additional level with the “do” arg to pants.
p
Awesome sauce. That could work very nicely.
👍 1
c
Almost works… some io assumption not holding up, as it seems:
Copy code
$ ./pants do hello
Hi from invoke task!
Traceback (most recent call last):
 ...
invoke.exceptions.ThreadException: 
Saw 1 exceptions within threads (OSError):


Thread args: {'kwargs': {'echo': None,
            'input_': <_io.TextIOWrapper name='<stdin>' mode='r' encoding='utf-8'>,
            'output': <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>},
 'target': <bound method Runner.handle_stdin of <invoke.runners.Local object at 0x10f2947c0>>}
...
    return os.getpgrp() == os.tcgetpgrp(stream.fileno())

OSError: [Errno 25] Inappropriate ioctl for device
h
@curved-television-6568 try with 
--no-pantsd
. IPython fails with this same error when using Pantsd 😞 One of our most egregious broken windows
🎯 1
c
Spot on! Thank you @hundreds-father-404 So, with this change:
Copy code
[cli.alias]
do = "--no-pantsd run :invoke --"
It works nicely.
Copy code
$ ./pants do hello
Hi from invoke task!
p
cool
7 Views
Open in Slack
PreviousNext
Powered by