https://pantsbuild.org/ logo
#development
Title
# development
w

wide-midnight-78598

03/19/2023, 5:49 AM
Worked on a slightly different approach to Pants bash completions this evening. I was wondering how usable completions that called back into Pants dynamically would be (eliminating the "completions per repo" problem). Turns out, they're incredibly unusable on my machine, because the
pants help-all
time is just too slow. Instead, for those who have
jq
installed, I've tried yet more slightly different. I call
pants help-all
and cache the resulting JSON using the md5 of the pants.toml file. I then check that md5 on completion, and pull the giant json file back in to use
jq
to parse. This won't work for nested
pants
usage, unfortunately. Also, this ends up being too slow to use for parsing all the scoped options from all the keys in the JSON (at least, using a naive for loop approach - I think
jq
has a syntax for what I'm trying to do, but I haven't dug into the manual yet). Either way, for this approach, it's probably worth just doing a one-time parse and caching all the results somewhere, until the pants.toml MD5 changes, and then it would be super fast (rather than re-parsing a huge JSON file all the time). https://gist.github.com/sureshjoshi/cbe50559eb85f21f079e45681d9b486b
I think if we had a fast enough runner or daemon, what we could do instead is call back into Pants using a
--pants-completion-helper
call or something, which could dynamically return appropriate values back to the completions. Another option is keeping this generic-style completion file, and when Pants runs, it automatically exports a parsed help-all json to a Pants-controlled location per repo, and the bash script would know by convention to look there 🤷
f

fast-nail-55400

03/19/2023, 10:04 AM
Either way, for this approach, it's probably worth just doing a one-time parse and caching all the results somewhere, until the pants.toml MD5 changes, and then it would be super fast (rather than re-parsing a huge JSON file all the time).
Which was the approach taken by the Pants v1 shell completion support. The options config was converted to be shell code and then the shell completion function could just consult shell variables to determine what the completions were.
w

wide-midnight-78598

03/19/2023, 1:29 PM
Ooh, that's interesting - I never knew that. I mean, the hope would be that we could get the load time down - but I think as John mentioned, we'd still need to import the world. Would just be nice to only do that once. My hope would still be complete for a
pants-completion-handler
idea. I think that would be slick, and extensible, and not require any 3rd party tooling
I'm planning on adding targets and a few other things to this - but it's all handled by calling
pants
at some point
p

proud-dentist-22844

03/20/2023, 5:37 PM
Is there an API the completions stuff could use to query pantsd?
w

wide-midnight-78598

03/20/2023, 6:08 PM
@witty-crayon-22786 ^^ ? I know unfortunately little about the daemon
Need to ++ my knowledge on that
w

witty-crayon-22786

03/20/2023, 6:14 PM
Is there an API the completions stuff could use to query pantsd?
`pantsd`’s only interface at the moment is CLI invokes. the actual runtime of: 1. starting the native client from https://github.com/pantsbuild/pants/pull/11922 2. connecting to a server … should be sub 50 ms though. most of the overhead when using 11922 is in options parsing and bootstrapping.
the difference between ~50 ms and the ~180 ms that @enough-analyst-54434 shows there is in options parsing / bootstrapping
f

fast-nail-55400

03/20/2023, 7:45 PM
`pantsd`’s only interface at the moment is CLI invokes.
It need not be given the Build Server Protocol support demonstrates Pants can function as a request/response server as well.
w

witty-crayon-22786

03/20/2023, 7:46 PM
Well, sure. But that's still using a CLI invoke.
f

fast-nail-55400

03/20/2023, 7:46 PM
A similar approach to BSP could be taken.
pantsd
listens on a UNIX domain socket under BUILDROOT/.pants.d for requests and then invokes an uncached rule to get the result.
so the UNIX domain socket part (for ease of service discovery and security) would be the difference
(you are correct that the BSP server is technically still a CLI invoke just enough to then run the BSP/LSP protocol)
(but no need to require a pantd API be setup that way)
w

witty-crayon-22786

03/20/2023, 7:48 PM
Yea, maybe. I don't expect that adding a new protocol would be faster. The majority of the (unavoidable) latency is in starting the client process, and in options parsing / etc.
f

fast-nail-55400

03/20/2023, 7:49 PM
well it need not me invoked on every completion request. a better system would have the completion support recognize when it has changed into a new/different Pants repository, query the pantsd for the completions for that repo, and then cache them as shell code once
like what
direnv
and similar tools do
or Pants could just write the shell code for the completions under
.pants.d
and completion support could just pick it up directly without any API or other extra stuff
really no need for an API per se to get completions per Pants repo if the shell code for completions was written out already
so two parts to completions support - the completion handler shell functions sourced in shell startup file, and then that shell function sources per-repo config whenever directory changes into a new repo.
and maybe a Pants config option to have pantsd write out completions when it starts up (to avoid pantsd startup cost for those not using completions)
w

wide-midnight-78598

03/20/2023, 8:49 PM
It's worth noting that I toyed around with an idea (more or less like this), where the completions script was a shim that calls out to the Pants-installed python interpreter, to then run a script which performs completions without the use of
jq
- performance was perfectly adequate. The ideal world for that would be as you described, where
pantsd
(or whatever) automatically writes out the completions, or caches help, whenever the pants.toml changes. I think that was the hope in one of the more recent tickets on this topic. I think back in the day, I also talked about the idea of the
./pants
script auto-sourcing a completions script as well - which was a bit more 🤷 .
2 Views