Improve search on website

[etiennebacher]

The search doesn't take into account the function names in the sidebar in "Reference", which makes the search experience a bit useless. For example, searching "sum_horizontal" doesn't point to pl$sum_horizontal() but only shows the output when we print all methods in one of the examples:

[etiennebacher]

The problem is that the search index is based on the md files, not the HTML ones: in the md file, pl_sum_horizontal exists, but not simply sum_horizontal. In addition to that, it can't search for chunks of words (for example searching for "ackage" doesn't suggest the word "package").

Searching for "pl_sum_horizontal" works fine (but is clearly not ideal because we'd need to prefix lazy functions with "LazyFrame_", etc.):

[etiennebacher]

That is something to fix in altdoc anyway because the search index is built when we run render_docs().

Edit: not sure of that actually, the problem is that the "Usage" section is originally really weird because of polars namespace

[etiennebacher]

This is annoying, the website loses half of its interest if search is impossible (or only with the trick of searching for e.g "DataFrame_"). To summarize the comments above, the problem is that the search index used by mkdocs is built using the markdown files (that are then converted to HTML by mkdocs). This is why we need to type "DataFrame_" to search in DataFrame methods: the "altdoc postprocessing" script that fixes the Usage section by going from "DataFrame_" to "<DataFrame>$" comes after the search index is built.

We may think that this can be fixed by running the "altdoc postprocessing" script earlier in the workflow, basically after every devtools::document(). However, this is not correct because R CMD check will error when it parses the ("fixed") Rd files (cf #701).

Therefore, one solution would be to do this step only in the altdoc workflow. This is what it would look like:

• make a copy of the "man" folder (with the original Rd files) named "man_old"
• run the script in build: add functions to update usage sections in Rd files #701 on the "man" folder. This fixes the formatting in the Usage section.
• run the altdoc workflow (preprocessing script + render_docs()). Since it runs on the fixed "man" folder, the search index should be built correctly (typing "DataFrame$" in the search bar should point to DataFrame methods)
• remove the "man" folder, and rename "man_old" to "man". Therefore, the "man" folder is now the same as what was generated by devtools::document()

About accessing docs inside R with ?: I think this cannot be solved. This uses the Rd files packaged with polars, and those have to pass R CMD check, which means they cannot be reformatted. This new workflow wouldn't change that.

Note that the solution probably shouldn't come from altdoc itself. polars is a very very special case, and I don't want to make big changes to altdoc just because of polars weird docs.

@eitsupi I'd like to have your opinion on this. Does this workflow make sense to you? Do you have other ideas?

[eitsupi]

@etiennebacher Thanks for the detailed explanation. I think the solution is reasonable.

[eitsupi]

@etiennebacher Any update on this?
It is quite troubling that the search is almost useless.
For example, when searching for "strptime".

[etiennebacher]

I had started working on this but I don't think the search was better in the end. My progress is here: https://github.com/pola-rs/r-polars/tree/rework-docs-workflow

I'll explore more, but we might have to wait for squidfunk/mkdocs-material#6307

[eitsupi]

Thanks for working on that.

My other question was whether it might be worth moving from Mkdocs Material to Quarto, as Mkdocs Material is very powerful but has a lot of proprietary notations that would benefit this repository less.

[etiennebacher]

I'm not convinced it's worth moving to Quarto. I can try to make a small demo when I have time, but from what I can see on the marginaleffects website the search is also not great. Typing plot_pred to search for the man page of plot_predictions() returns tons of results, and it's not clear at all which page is the man page.

has a lot of proprietary notations that would benefit this repository less

Not sure I understand, what do you mean by proprietary notations?

[eitsupi]

it's not clear at all which page is the man page.

There is no doubt about this.

Not sure I understand, what do you mean by proprietary notations?

If we can use these, the website will look much better.
https://squidfunk.github.io/mkdocs-material/reference/

[etiennebacher]

If we can use these, the website will look much better.
https://squidfunk.github.io/mkdocs-material/reference/

I don't see obvious features that would make the website better, but if you list some I can take care of including them

[squidfunk]

Author of Material for MkDocs here! We'd be happy to get you guys as a beta tester once we have the next search preview available. In the meantime, you can already test the research preview 2, which should improve accuracy. We really would love more feedback on this, because we finally want to build a search that doesn't suck ☺️ Additionally, for some more context see squidfunk/mkdocs-material#6307, where we explain the why and what we've planned.

[eitsupi]

Sounds great, thanks for your comment @squidfunk!

I think it is really great that Material for Mkdocs provides a great local search feature out of the box.
(I have built Japanese documents with Docsaurus and had quite a bit of trouble setting up a local search feature for Japanese)

[etiennebacher]

@eitsupi FYI I'm quite happy with the recent improvements from #925 so I'm not actively looking into this issue anymore