https://pantsbuild.org/ logo
#development
Title
# development
w

wide-midnight-78598

04/07/2023, 2:54 PM
Who would know how the pants docs search engine is loaded? I feel like https://www.pantsbuild.org/docs/reference-global isn't being indexed correctly. I've never really had success searching items on there
Screenshot 2023-04-07 at 10.53.57 AM.png
p

proud-dentist-22844

04/07/2023, 3:01 PM
I've had issues finding the global options as well.
w

wide-midnight-78598

04/07/2023, 3:04 PM
Yeah, definitely not running on Algolia 🙂 I've never really known how readme search works actually - Not sure if it's a scraper (e.g. maybe we're not correctly linking to the global options page?) or whether it should ingest all the HTML
p

proud-dentist-22844

04/07/2023, 3:13 PM
We need something that indexes symbols. Searching for
__defaults__
doesn't work because it ignores the underscores.
I wonder if sphinx + rtd would fare any better
w

wide-midnight-78598

04/07/2023, 3:19 PM
Searching for
__defaults__
doesn't work because it ignores the underscores.
Oh, wild, I didn't know that. Though in hindsight, a lot of discovery problems for me are making sense 🤦‍♂️
b

busy-vase-39202

04/07/2023, 3:55 PM
Its search is poor, unfortunately shifting more tech support burden onto the team than should be necessary. Anyone who'd like to take on the work of moving us over to sphinx would be doing a great service.
w

wide-midnight-78598

04/07/2023, 4:04 PM
Not sure if that would help searchability - unless the hosting platform would?
Maybe if we could get an Algolia open source plan - I vaguely recall they had/have/will have a site scraper - so you kinda just point it at your docs site... Though, maybe I'm conflating which search engines I was looking at
b

busy-vase-39202

04/07/2023, 4:06 PM
One of the many issues with readme is that we are not allowed to use custom CSS or custom JS, which severely restricts our ability to use external tooling. We could link to an external indexing tool, but we could not integrate it.
w

wide-midnight-78598

04/07/2023, 4:07 PM
Oh wow, didn't know that.
b

busy-vase-39202

04/07/2023, 4:07 PM
And if all we can do is link to a custom search engine, might as well use something like Google Custom Search for free.
There is a tier that allows custom JS, but it's very expensive; well beyond the community's means.
w

wide-midnight-78598

04/07/2023, 4:09 PM
I statically host most of my sites on Cloudflare pages - was just going through the process of adding Algolia to my blog. Also I use Algolia for an enterprise product. Not cheap, but so good
p

proud-dentist-22844

04/07/2023, 4:09 PM
Sphinx, restructured text, and markdown are on my list of things to work on in pants, but there are several other things I need to do before I’ll get to that. So, if anyone gets there before me, please! 🙂
🎉 2
b

busy-vase-39202

04/07/2023, 4:12 PM
Just getting us moved over to sphinx would be a big service, since it would give us a lot more flexibility in taking next steps. As far as I know, most of the the docs are already in markdown since it's the format readme supports.
One thing that's a bummer to me about readme's indexing is that it only indexes readme itself. blog.pantsbuild.org and chat.pantsbuild.org for instance are not indexed, nor is GH. Imagine if all of that content were indexed in one place. People would be able to do so much more self-service support, and keep moving quickly.
readme also only indexes certain content. The home page, for instance, is not indexed. We also are restricted in what we can do with the theme, so for instance the reason the header menu is overpopulated with stuff like ToS, privacy policy, and cookie policy because we can't create a footer.
w

wide-midnight-78598

04/07/2023, 4:19 PM
I feel like a lot of questions I've been asking myself are slowly being answered.....
b

busy-vase-39202

04/07/2023, 4:20 PM
Pages like https://www.pantsbuild.org/docs/who-uses-pants and https://www.pantsbuild.org/docs/language-support exist purely because of readme's awkward architecture constraints.
Every time I see those "click here" to go to the actual page, I'm like grrr, that is so 1996.
A lot of FAQs we get are because the search is fairly primitive. For instance, "rename build files" and "name build files" do not turn up any obvious link to the info the person is seeking.
p

proud-dentist-22844

04/07/2023, 4:27 PM
Is the blog made with markdown stored in the pants repo too? Any indexing sphinx does is of the content it generates.
w

wide-midnight-78598

04/07/2023, 4:28 PM
b

busy-vase-39202

04/07/2023, 4:29 PM
It also doesn't handle decimals correctly. So searching for a decimal number like 3.8 results in matches for anywhere a 3 appears, regardless of whether it's followed by dot-eight.
p

proud-dentist-22844

04/07/2023, 4:29 PM
another case of using a generic text search engine that strips out symbols
b

busy-vase-39202

04/07/2023, 4:30 PM
Yeah. For a tool that's meant to be specialized for handling code, readme is disappointingly not well-tuned to the task.
It seems to discard many symbols, so for instance, searching for
./pants
gets 21 pages of matches that mostly are not actually matches.
p

proud-dentist-22844

04/07/2023, 4:32 PM
yup. not surprised.
b

busy-vase-39202

04/07/2023, 4:35 PM
In answer to your earlier questions, stuff that isn't autogenerated is in
<https://github.com/pantsbuild/pants/tree/main/docs/markdown>
btw.
p

proud-dentist-22844

04/07/2023, 4:35 PM
including the blog?
b

busy-vase-39202

04/07/2023, 4:35 PM
No, the blog is hosted with Ghost
And chat.pantsbuild.org is hosted by Linen. So neither is part of the repo.
p

proud-dentist-22844

04/07/2023, 4:37 PM
ah. Then yeah - sphinx and rtd won’t help with a combined search for all of those. Or at least not out of the box. We would need something else that indexes all three, which we could wire up through sphinx.
w

wide-midnight-78598

04/07/2023, 4:37 PM
I think not being able to search slack/linen makes sense to some degree. Ditto for those communities using Discord. Until you're in the app, searching is hard, unless they have some API that allows digging in?
p

proud-dentist-22844

04/07/2023, 4:37 PM
linen is designed to be search-engine friendly
b

busy-vase-39202

04/07/2023, 4:38 PM
Well if we had access to custom JS and custom CSS, then we could integrate any open source or low-cost search tooling we want.
👍 1
w

wide-midnight-78598

04/07/2023, 4:38 PM
linen is designed to be search-engine friendly
Really? I didn't see it come up anywhere in search results when doing exact quotes... Hmm
p

proud-dentist-22844

04/07/2023, 4:39 PM
Well, that’s one of the things they advertised I thought. 😕
b

busy-vase-39202

04/07/2023, 4:39 PM
Linen isn't yet indexed by Google et al. Until we publish more links to it for search engines to find, it's still pretty low-profile. But its own search is pretty good.
👍 1
w

wide-midnight-78598

04/07/2023, 4:40 PM
site:<http://chat.pantsbuild.org|chat.pantsbuild.org> pants
I was able to get some stuff showing up, which is pretty neat! I'd say it's not well indexed yet, since the only stuff that popped up was the top-level messages, not threads, but maybe it'll get better as it goes on
🚀 1
b

busy-vase-39202

04/07/2023, 4:42 PM
I'm impressed it's found it at all. The first public link was in the blog post a couple days ago. It may have only just begun spidering, which would explain the depth=1 results.
w

wide-midnight-78598

04/07/2023, 4:44 PM
Let Google answers questions and not your team
Linen is designed to be search-engine friendly. Most if not all chat tools are not search-engine friendly or accessible. Linen is search-engine friendly because we render a static version of our app to search engines like Google
Love it so much. It's annoying to try to search through a team's discord or slack (but first needing to sign up) - walled garden approach to open-source support. Github issues + discussions? No problem, great results. But I just think about the thousands of questions asked every day on Svelte's discord that would be so helpful to see when ~googling~DuckDuckGo-ing. Another side effect of all the chat systems is that StackOverflow really isn't the go-to resource when you're looking for something framework/library specific, which is also unfortunate.
2
b

busy-vase-39202

04/07/2023, 4:47 PM
We put 8 years of Slack history into Linen, so I'm guessing there are a lot of static pages to crawl.
🎉 1
👖 1
b

broad-processor-92400

04/07/2023, 9:18 PM
FWIW (a lot, I think) I was searching for something a few days ago and turned up some old and vaguely useful results buried in a slack thread on linen 👍
🚀 2
c

curved-television-6568

04/08/2023, 1:02 PM
How about getting our Pants docs onto devdocs.io ? https://github.com/freeCodeCamp/devdocs/blob/main/.github/CONTRIBUTING.md#contributing-new-documentations I’ve used devdocs in the past and it has mostly great searchability for the stuff on there (with a few exceptions)
It’s a docs scraper, so doesn’t mean we host the docs any where else, just that it is scraped from where ever and shows up there as well.
👀 1
5 Views