https://pantsbuild.org/ logo
#general
Title
# general
p

proud-dentist-22844

07/23/2021, 4:43 PM
Another feature idea (possibly in a plugin): There are a few developer tasks that don't fit within the pants goals:
fmt
,
lint
,
package
,
repl
,
run
,
test
, and
typecheck
(and friends: https://www.pantsbuild.org/docs/reference-all-goals) For instance there are a bunch of scripts here: https://github.com/pantsbuild/pants/tree/main/build-support/bin And other projects will have their own set of ad-hoc tasks that need to be done (regen constraints, or build a venv for external tooling, or version bumping, or ...) With a
Makefile
, these tasks get added as additional custom make targets. We could use pants plugins to add more goals, but a pants-plugin feels like overkill for some of these tasks that are not meant to ever pass through remote caching & execution. But I really want
./pants
to be the single point of entry for developer tasks. So, what if there was a lighter weight way to extend pants? Maybe an
invoke
pants-plugin (or call it a
do
plugin?) that allowed for running tasks written for http://www.pyinvoke.org/ ? Essentially, a way to have a collection of plain (or nearly plain) python scripts that pants can run locally? Maybe the plugin would be able to dynamically create
--help
for all of the available scripts so that it's a bit more structured than
./pants run
, but still extremely flexible.
✔️ 2
h

hundreds-father-404

07/23/2021, 4:48 PM
Interesting idea! What would be your ideal UX for how you invoke those custom scripts?
./pants custom-script
(goal syntax) or something like
./pants run path/to/custom_script.py
p

proud-dentist-22844

07/23/2021, 4:49 PM
./pants custom-script
or
./pants do custom-script
(something really short)
👍 1
Maybe
./pants invoke custom-script
but that starts to feel less accessible (only 4 more chars, yes, but then why not just do
invoke custom-script
? I want a way to have
./pants
be the entry point even if it uses invoke or something else to enable the feature).
Context for my thoughts: My work moving StackStorm to pants overlaps with this: https://github.com/StackStorm/st2/pull/4782 (replace Makefile with invoke). But not all of these tasks fit neatly into one of the pants goals. So, I was trying to think of a way to reuse parts of this with pants. Thus the feature idea.
f

fast-nail-55400

07/23/2021, 5:19 PM
should this be similar in spirit to
npm run
?
👍 1
🤔 1
i.e., just have a
[scripts]
section in pants.toml for each of these scripts
c

curved-television-6568

07/23/2021, 5:37 PM
there’s an example in the pants docs about writing a “binary run plugin” that executes .. stuff. Not entirely sure of its context, though (what it runs), just came to think of it. Also, along with the parallell to
npm run
is
grunt
with it’s tasks, etc.. just for inspiration.. 😉 Edit: this doc: https://www.pantsbuild.org/docs/plugins-run-goal
p

proud-dentist-22844

07/23/2021, 6:04 PM
Doesn't
./pants run
require a path to the script to run? That doc looks like it just enables new kinds of binaries to run via
./pants run
, right? But you still have to know the path to the script to run it. Does that offer any benefit over running the script directly?
h

hundreds-father-404

07/23/2021, 6:06 PM
Doesn't ./pants run require a path to the script to run?
Yes
hat doc looks like it just enables new kinds of binaries to run via ./pants run, right?
Yeah, that's correct. That guide is meant as a guide for how you might do something like get
./pants run script.sh
to work -- It sounds like what you're after is a way to add new custom goals like
./pants release
or
./pants changelog
, but in a more accessible way than the Plugin API
p

proud-dentist-22844

07/23/2021, 6:07 PM
yeah. And that more accessible way doesn't need access to the rules or targets (or other pants) APIs.
h

hundreds-father-404

07/23/2021, 6:10 PM
My first intuition is: make the rules API simpler such that it is accessible, including: - fix rule graph errors - better docs and examples Prefer to improve our main API rather than bifurcating into more APIs. But it may be the case that it's still Too Much and we need a more limited API, which we should be open to imo
p

proud-dentist-22844

07/23/2021, 6:10 PM
should this be similar in spirit to 
npm run
?
That might be an interesting extension to
./pants run
where you can register some "shortcut targets" in pants.toml and then use
./pants run <shortcut target>
to run the script without having to pass in the absolute path or full target.
The problem with using the rules API is that I'm supposed to think of it as a DSL, not raw python.
✔️ 1
h

happy-kitchen-89482

07/24/2021, 1:02 AM
Basically a generic goal to run scripts?
But then you have to take extra care around side effects and such?
p

proud-dentist-22844

07/24/2021, 1:09 AM
It would be explicitly side-effecty
and therefore could not run via remote execution
So, yeah - it's sort of like genrule, but that sounds like a rule that could get triggered in the normal course of the other goals. I'm thinking more about things that would only run when explicitly requested.
q

quaint-gold-40000

07/26/2021, 10:08 AM
I've found https://pydoit.org/ to be great for describing and running generic tasks.
👀 1
h

happy-kitchen-89482

07/26/2021, 6:29 PM
@witty-crayon-22786 Can we somehow mark processes as not-remotable today?
w

witty-crayon-22786

07/26/2021, 6:31 PM
Only via InteractiveProcess, which also forces it to run in the foreground, and is only legal in a goal_rule
But that's exactly where you'd want "things that only run when explicitly requested"
h

happy-kitchen-89482

07/26/2021, 7:57 PM
And "things that can side-effect"
So @proud-dentist-22844 the idea is that no rules depend on the output of running these processes? Running them is the end goal itself?
p

proud-dentist-22844

07/26/2021, 7:59 PM
yup
h

happy-kitchen-89482

07/26/2021, 8:05 PM
Then this should be pretty straightforward
Want to take a swipe at it? 🙂
p

proud-dentist-22844

07/26/2021, 8:07 PM
Sure. Next time I'm hacking on st2+pants, I'll work on this 🙂
Is there something like
InteractivePexProcess
? I build an
invoke
pex, and then I need to make an
InteractiveProcess(..., run_in_workspace=True)
but
InteractiveProcess.from_process()
doesn't have a
run_in_workspace
arg. https://github.com/pantsbuild/pants/blob/main/src/python/pants/engine/process.py#L328 Is there a way to run a pex interactively in the workspace?
Copy code
InteractiveProcess(
            argv=invoke_pex.argv,
            env=invoke_pex.env,
            input_digest=invoke_pex.input_digest,
            run_in_workspace=True,
            forward_signals_to_process=True,
        )
I think that's it
Except that you can't use input_digest and run_in_workspace together.
hmm
https://www.pantsbuild.org/docs/rules-api-installing-tools#pex-install-binaries-through-pip has an issue: it should use
InterpreterConstraints
instead of
PexInterpreterConstraints
.
Looks like there are a couple other thing that are out-of-date with that
PexRequest
in the docs.
main
instead of
entry_point
and
main
has to be a
MainSpecification
.
Here's a skeleton for running invoke in the workspace: https://github.com/st2sandbox/st2/blob/pants/pants-plugins/do_invoke/goal.py Next steps: • figure out how passing args to from commandline to InteractiveProcess works • figure out how to get pants to pull --help details from the subprocess to merge, UX-wise, the --help output.
w

witty-crayon-22786

07/31/2021, 8:55 PM
figure out how passing args to from commandline to InteractiveProcess works
this is done via what is known as “passthrough” args on a subsystem: https://www.pantsbuild.org/docs/rules-api-subsystems#passthrough
figure out how to get pants to pull --help details from the subprocess to merge, UX-wise, the --help output.
this will be challenging, because options setup can’t run processes… you’d have to do it statically
p

proud-dentist-22844

08/01/2021, 7:02 AM
Hmm. I might have to embed
invoke
in the pants plugin instead of using a pex so that the integration can feel more seamless.
What do you think of extracting the available options like this? https://github.com/st2sandbox/st2/blob/pants/pants-plugins/do_invoke/goal.py#L50-L102
h

hundreds-father-404

08/05/2021, 10:34 PM
Huh, interesting. Very creative! Are you envisioning this will be something like
./pants do --cmd1 --cmd2
? I'm being a little lazy not looking more directly into invoke's source code: do you expect that code to be stable deterministic? To be sure, it doesn't depend on the argv you used to run
./pants
?
p

proud-dentist-22844

08/05/2021, 10:44 PM
I expect it to depend only on what tasks are available (invoke looks at the file system). It could depend on argv, but I bypassed all that stuff because I want something more deterministic. I guess the presence of tasks is similar to registering pants backends to enable additional commands.
👍 1
h

hundreds-father-404

08/05/2021, 10:48 PM
Cool. That's good. I suspect this code will not properly handle file watching, meaning if you're using pantsd and change the invoke files, it won't be picked up If that is indeed a problem, an awkward workaround is to add those files to
pantsd_invalidation_globs
in
pants.toml
under
[GLOBAL]
The options system does not in general wire up to file watching, tracked by https://github.com/pantsbuild/pants/issues/10360 but otherwise, I think this works!
👀 1
Hmm. I might have to embed invoke in the pants plugin instead of using a pex so that the integration can feel more seamless.
I don't totally follow that. Is the concern the performance to create the Pex the first time?
Btw, I like the modeling of this being options on the
do
goal. It keeps
./pants help goals
tighter, allows you to see all the options w/
./pants help do
p

proud-dentist-22844

08/06/2021, 2:28 AM
Is it possible to have non option arguments after the goal? like
./pants do task1 --task1-option
?
f

fast-nail-55400

08/06/2021, 7:44 AM
Yes, there is a notion of
passthru
(pass through) args where any arg after
--
on the command line is a pass-through arg.
👍 1
so you would do:
./pants do task1 -- --task1-option
then
--task1-option
should be available via
.passthru
on one of the options instances
I’d like to say on the
GoalSubsystem
subclass for your
do
goal but this is where my knowledge is lakcing
h

hundreds-father-404

08/06/2021, 1:56 PM
Look at black/subsystem.py as an example of pass through args
p

proud-dentist-22844

08/06/2021, 3:28 PM
No passthrough args doesn't do what I'm looking for - I want to add a list of "sub" goals. But registering an option requires
--
or
-
prefix, doesn't it? Is there another way to register a positional argument?
h

happy-kitchen-89482

08/09/2021, 6:34 PM
There isn't atm - all cmd-line flags must be -x or --long-name
Why wouldn't passthru work? Is it that you don't like the floating
--
in there?
So it would be
./pants do -- task1 --task1-option
p

proud-dentist-22844

08/09/2021, 9:12 PM
No, I don't like the floating
--
and I'm working on dynamically registering all the options so that they show up in pants' help output.
Option registration should be working fine. A way to register sub-goals would be nice though. Ooh. Thinking about sub-goals, it would also be nice to target particular linters:
./pants lint black
or maybe use something other than space to say "this is a subgoal"?
./pants lint.black
h

hundreds-father-404

08/09/2021, 9:20 PM
Pants v1 used to have something similar to that, which is what Benjy just finished removing hehe It's plausible we'd want to add back but in a re-envisioned way
f

fast-nail-55400

08/09/2021, 9:24 PM
is there any reason to not just defer to the sub-tool for option parsing of tool-specific options?
p

proud-dentist-22844

08/09/2021, 9:27 PM
To use pants as a seamless single-point-of-entry for dev tasks.
Maybe I'll have to add an extra
./invoke
entry point (or
./do
) that uses pants to setup the tool, and then run it. Not having to create a virtualenv (or the makefile to orchestrate that) to get up and running is really nice.
👍 1
Then there's more than one entry point, but the tool designs seem to conflict, so merging them is not working out.
h

hundreds-father-404

08/09/2021, 9:33 PM
This could be a bad idea, not sure I recommend it. Another option is to update
./pants
bash script to detect if
./pants do
is used and sanitize it to Pants options
p

proud-dentist-22844

08/09/2021, 9:33 PM
hmm. Then updates for that script would become more difficult.
1
(lol - maybe one of the adhoc tasks could be to update the
pants
script)
h

hundreds-father-404

08/09/2021, 9:35 PM
Yeah, and we do want to move to distributing Pants as a binary rather than PyPI (altho possibly the bash script would still exist, unclear)
p

proud-dentist-22844

08/10/2021, 12:32 AM
GIven a 3rd party dep in pants.toml >
[GLOBAL].plugins
, can I query pants (from the cli in bash) for the path to a virtualenv / pex that includes that requirement? Or the PYTHONPATH I should use to access it?
h

hundreds-father-404

08/10/2021, 2:21 AM
Hmm @witty-crayon-22786 do you know by chance?
w

witty-crayon-22786

08/10/2021, 3:07 AM
hm, no… that’s a well kept secret / implementation detail, heh. plugins are resolved using PEX, but then referenced directly out of the PEX cache.
p

proud-dentist-22844

08/10/2021, 3:10 AM
well, here's my invoke plugin v2, using a wrapper script: The wrapper script (accessible as
./invoke
or
./inv
or (my favorite)
./do
) https://github.com/st2sandbox/st2/blob/pants/invoke And the pants-plugin with the magic wrapper script: https://github.com/st2sandbox/st2/blob/pants/pants-plugins/do_invoke/goal.py#L144-L182 (rough, but functional)
(I'm probably relying on a bunch of implementation details I shouldn't... lol)
I'm open to suggestions and yelling (things like "What on earth are you thinking!?" might be appropriate 😉 )
😆 1
@witty-crayon-22786 How safe/wise is creating this script: https://github.com/st2sandbox/st2/blob/pants/pants-plugins/do_invoke/goal.py#L75-L77 Which relies on the plugin here: https://github.com/st2sandbox/st2/blob/pants/pants.toml#L37 and gets called here: https://github.com/st2sandbox/st2/blob/pants/invoke#L16-L24 Am I shooting myself in the foot by doing this? Once pants becomes a binary, it will still need some kind of external path to hold
[GLOBAL].plugins
, so I believe I can rely on those plugins being available on
sys.path
somewhere, right?
w

witty-crayon-22786

08/10/2021, 8:28 PM
Once pants becomes a binary, it will still need some kind of external path to hold 
[GLOBAL].plugins
 , so I believe I can rely on those plugins being available on 
sys.path
 somewhere, right?
yea… any plugin loading mechanism will involve them being added to
sys.path
, so that should continue to work afaik
h

happy-kitchen-89482

08/12/2021, 2:18 PM
For the record, what I removed wasn't "subgoals", which we never had in v1, but just subscoping of options. So when you ran
./pants lint.flake8 --foo
it actually ran the entire
lint
goal, for all linters, but allowed
--foo
to refer to just the
lint.flake8
scope. What @proud-dentist-22844 is proposing is different I think?
👍 1
p

proud-dentist-22844

08/12/2021, 2:34 PM
Wait is
lint.flake8
a thing?
Could that be repurposed for subgoals? To run only the flake8 linter?
w

witty-crayon-22786

08/12/2021, 3:47 PM
there was talk of having a way to do that… basically
./pants lint --only=flake8 ..
p

proud-dentist-22844

08/12/2021, 3:49 PM
yup.
lint.flake8
would be a nice way to target “subgoals”, essentially saying that each linter is a subgoal of
lint
. Then, I could use those subgoals elsewhere too.
If we didn’t do that, then
--only
is the next best experience imo.
h

hundreds-father-404

08/12/2021, 3:55 PM
What would multiple linters look like?
./pants lint.flake8 lint.isort
? vs
./pants --lint-only=isort,flake8
probably
p

proud-dentist-22844

08/12/2021, 4:05 PM
Is it possible to do multiple goals right now?
./pants lint fmt test
? If so, then
lint.flake8 lint.isort fmt.black
could make sense.
h

hundreds-father-404

08/12/2021, 4:06 PM
Yeah it's possible to do
./pants fmt lint test
!
p

proud-dentist-22844

08/12/2021, 4:06 PM
And if multiple goals isn’t possible at this point, maybe checkout the UX from
invoke
for multiple goals + their options.
oh, it is? cool
With invoke, options are positional. Translating that to pants would be something like:
./pants fmt <options for fmt> lint <options for lint> test <options for test>
h

hundreds-father-404

08/12/2021, 4:07 PM
(We have a TODO to allow those rules to run concurrently, which is an interesting problem because you have to keep track of things like how
fmt
can overwrite files and invalidate the rest)
p

proud-dentist-22844

08/12/2021, 4:11 PM
The
<goal>.<subgoal>
would enable some cool shell shortcuts.
./pants lint.{flake8,isort,black} fmt.black
w

witty-crayon-22786

08/12/2021, 4:32 PM
With invoke, options are positional. Translating that to pants would be something like: 
./pants fmt <options for fmt> lint <options for lint> test <options for test>
they are in Pants in general as well… we used to have a page explaining this, but any options shown by
./pants help $goal
can be used positionally with that goal
(the page still exists for v1, and ironically, it’s called: “Invoking Pants”: https://v1.pantsbuild.org/invoking.html)
we also need to document the
--loop
option, which watches the filesystem and re-runs your goals. when i’m iterating on a test, i usually use:
Copy code
./pants --loop test typecheck fmt lint $test
p

proud-dentist-22844

08/12/2021, 6:12 PM
Hmm. It also needs to be documented in the
--help
output somehow. I've been looking at that and assumed I had to always be extremely verbose:
--lint-*
@busy-vase-39202 I guess I really do rely on the
--help
output more than I thought. LOL
b

busy-vase-39202

08/12/2021, 6:28 PM
Haha.
w

witty-crayon-22786

08/12/2021, 6:40 PM
it’s in
./pants help global
… bare
./pants help
doesn’t show much at all
(…although it says it’s “alpha” which is no longer the case. will fix.)
c

curved-television-6568

10/10/2021, 7:21 AM
@proud-dentist-22844 Have you made any progress on the invoke stuff? I’ve been piecing together a few ideas in my head, and just realized it could be an interesting approach to for this topic on how to run “ad-hoc” scripts. Given the new
experimental_shell_command
(which is kind of like the Bazel
genrule
Benjy mentioned) if used with
./pants export-codegen
could work pretty well as the entry point to such scripts. Now to make it user friendly, I think that there could be a new “alias” subsystem, where you can configure cli arg replacements. I.e.
Copy code
[alias.affected]
args = "--changed-since=origin/boss --changed-dependees=transitive"
Then you may use that as
./pants affected test
So with this, we could infer aliases based on shell command targets in, say a scripts directory.
Copy code
/scripts/some-task.sh

#scripts/BUILD
experimental_shell_command(name="some-task", command="./some-task.sh", outputs=[…])
Then
./pants export-codegen scripts:some-task
could be aliased to
./pants do some-task
or even
./pants some-task
This has some cool side effects, like that you can depend on other targets for your tasks, etc. Also, if we add a output_path to the shell command, it should be allowed to replace/update/create files in the workspace, rather than under
dist/
. Also, the task may be run by remote execution, just as any other target. In summary, we need support for aliases (which I think would be generally useful regardless of this idea) and some tweaks to the shell command. Would be happy to work on this, if there’s agreement that it’s a cool idea :)
👀 1
w

witty-crayon-22786

10/11/2021, 5:46 PM
the aliases concept is a neat one… there was a previous proposal in that general area (although for fairly different reasons!), but this is even shorter.
👍 1
regarding using `export-codegen`here though: it seems like maybe we might want to add
run
support for
experimental_shell_command
, or add a second target type (sharing a lot of the implementation), which is more parallel to the other
*_binary
types, and explicitly about being
run
in the foreground
💯 1
c

curved-television-6568

10/11/2021, 6:22 PM
Yeah, I like the idea of running a
experimental_shell_command
.
And for shell command run support: https://github.com/pantsbuild/pants/pull/13229
There’s no alias inference based on any experimental_shell_command targets yet.. not really sure how that would work well (and effectively). Feeling that it might be good enough, and clearer, to require any scripts to be added as alias in the configuration explicitly.
p

proud-dentist-22844

10/13/2021, 6:15 PM
OK. I took a look. Yes, I got my invoke goal working. https://github.com/st2sandbox/st2/tree/pants/pants-plugins/do_invoke It uses a bit of a hack to install invoke in the pants venv, and then run it from there to skip pex-ing anything and get it to run as fast as possible. As such, it does not work with remote execution, as I went with the assumption that invoke tasks are local-only / dev-only tasks.
👀 1
aliases would be really cool though 🙂
I’m not sure how codegen +
experimental_shell_command
+ aliases for invoke would look, but I would be interested in something along those lines for invoke.
c

curved-television-6568

10/13/2021, 6:26 PM
Interesting, so you use pants to create the invoke wrapper script, and then run
./invoke …
directly, rather than using
./pants invoke …
, right?
I think it would be pretty easy to wrap up invoke in a pex as a dependency to a shell command, and run that with pants, and aliases to make any invoke task runnable as
./pants do <invoke-task> <task args>
p

proud-dentist-22844

10/13/2021, 11:14 PM
No. Pants invokes the wrapper script as well. https://github.com/st2sandbox/st2/blob/pants/pants-plugins/do_invoke/goal.py#L97 but it doesn't need to recreate the wrapper script because that is cached.
ah. No nevermind. You're right. It's been too long since I looked at this.
Making pants aware of the invoke args proved quite problematic, so I opted to have
./invoke
be the entry point for that, and it uses pants under-the-covers to manage things.
f

fast-nail-55400

10/13/2021, 11:24 PM
Making pants aware of the invoke args proved quite problematic,
were there any particular obstacles?
p

proud-dentist-22844

10/13/2021, 11:39 PM
--
I either had to sacrifice exposing things in help, and use
--
, or I had to go through a lot of hoops to grab the dynamic list of tasks available by importing invoke from within the plugin.
Also, invoke has the concept of targets and sub targets, so I would have needed some way to create sub-goals, but pants doesn't have that.
I guess, to use the alias feature, you'd end up registering each invoke target as an alias to reduce the number of levels you have to explicitly add on the command line to get to the function you want.
👍 1
c

curved-television-6568

10/14/2021, 5:36 AM
I’m not yet familiar with python-invoke, but how about
Copy code
[cli.alias]
do = “run :invoke --“
So any args after
do
will be pass through args to invoke, so as I suggested above, this could look like
./pants do some-task task-arg …
. So it only takes one additional level with the “do” arg to pants.
p

proud-dentist-22844

10/14/2021, 2:07 PM
Awesome sauce. That could work very nicely.
👍 1
c

curved-television-6568

10/14/2021, 2:26 PM
Almost works… some io assumption not holding up, as it seems:
Copy code
$ ./pants do hello
Hi from invoke task!
Traceback (most recent call last):
 ...
invoke.exceptions.ThreadException: 
Saw 1 exceptions within threads (OSError):


Thread args: {'kwargs': {'echo': None,
            'input_': <_io.TextIOWrapper name='<stdin>' mode='r' encoding='utf-8'>,
            'output': <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>},
 'target': <bound method Runner.handle_stdin of <invoke.runners.Local object at 0x10f2947c0>>}
...
    return os.getpgrp() == os.tcgetpgrp(stream.fileno())

OSError: [Errno 25] Inappropriate ioctl for device
h

hundreds-father-404

10/14/2021, 2:33 PM
@curved-television-6568 try with
--no-pantsd
. IPython fails with this same error when using Pantsd 😞 One of our most egregious broken windows
🎯 1
c

curved-television-6568

10/14/2021, 2:37 PM
Spot on! Thank you @hundreds-father-404 So, with this change:
Copy code
[cli.alias]
do = "--no-pantsd run :invoke --"
It works nicely.
Copy code
$ ./pants do hello
Hi from invoke task!
p

proud-dentist-22844

10/14/2021, 4:38 PM
cool
6 Views