[ENH] Implement new scrollspy

[peytondmurray]

This PR implements a new scrollspy mechanism to highlight entries in the secondary sidebar TOC; fixes #1965, and builds on top of #2107.

Approach

Every link in the secondary sidebar TOC is a reference to a section in the main article. This PR works by adding an IntersectionObserver that observes the heading at the top of every one of these sections. As the user scrolls down, the IntersectionObserver triggers a callback when new section headings come into view; an object containing the visibility of the various sections is kept up to date by these events. The first visible heading is the one that needs to have its respective TOC entry highlighted.

Other Notes

• Unrelated, but I kept getting errors coming from a place in setupAnnouncementBanner where a const was being assigned to. I fixed this just to make it stop complaining. If you'd rather have this in a separate PR, I'm happy to do so.
• The IntersectionObserver is set to observe element crossings with the viewport, but because there's a sticky menu bar at the top of the page I added a rootMargin to pad down the top edge so that element crossings happen below that menu bar. Testing by hand this seems to work pretty well.
• I also tested clicking on entries in the TOC, and as long as you click something high enough up on the page (not close to the bottom), the correct element gets highlighted as expected. Of course, if you have a bunch of sections right at the bottom, the one you click on won't be highlighted:

     -------------
     | Section A |
     |           |
   --|-----------|---
   ⁞ |           |  ⁞
   ⁞ |           |  ⁞
   ⁞ |           |  ⁞
   ⁞ | Section B |  ⁞  <-- Viewport is still considered to be focused on Section A,
   ⁞ | Section C |  ⁞      which extends all the way to the start of Section B
   ⁞ | end       |  ⁞
   ------------------

[gabalafou]

This looks great! I haven't tested it yet, so I may have some more feedback but I thought you might appreciate me sharing my code feedback before waiting on me to test it.

[gabalafou]

I just now rebased to see if we can get a working preview build.

[peytondmurray]

Awesome, I'll get these comments addressed now 🚀

[github-actions]

Coverage report

This PR does not seem to contain any modification to coverable code.

[gabalafou]

Okay, I gave this a whirl on the read the docs preview build, and for the most part it's great!

There's just one thing. I know you're already aware of this, but I find it really annoying that I can sometimes click on a link in the right sidebar table of contents and sometimes (when the corresponding heading is near bottom of the page and too close to headings above) it doesn't highlight that link in the TOC after I click it. This doesn't match the pattern established in the rest of the site, where clicking a TOC link (left bar or header nav) highlights that link. This feels like a big enough pattern break and UX frustration to be a blocker. I think without this being fixed I would prefer merging my PR over this one, even if it removes scroll spying, simply because it gets the TOC link highlighting correct. What do you think? Is this something you want to fix? Would you like me to look into fixing it?

[peytondmurray]

Agreed, but what's the right way to handle this? I think with the current machinery, as the page scrolls down to the selected heading it will highlight the TOC elements you see as the viewport scrolls down. I can think of a few ways of doing this with unobserve/observe, but I think I'd start by trying a few methods and seeing what works.

[peytondmurray]

Ahh, I don't think we can merge this yet. Tests are not working for me locally, and I think CI is not running the playwright tests, looking through the logs. Here's the invocation from CI.yml:

      - name: "Run tests ✅"
        shell: bash
        run: |
          # this will compile the assets and translations then run the tests
          # check if there is a specific Sphinx version to test with
          # example substitution: tox run -e compile-assets,i18n-compile,py39-sphinx61-tests
          if [ -n "${{matrix.sphinx-version}}" ]; then
            python -Im tox run -e compile-assets,i18n-compile,py$(echo ${{ matrix.python-version }} | tr -d .)-sphinx$(echo ${{ matrix.sphinx-version }} | tr -d .)-tests
          # if not we use the default version
          # example substitution: tox run -e compile-assets,i18n-compile,py39-tests
          else
            python -Im tox run -e compile-assets,i18n-compile,py$(echo ${{ matrix.python-version }} | tr -d .)-tests
          fi

So CI runs for example the py312-sphinx61-tests tox command, which invokes coverage run -m pytest -m "not a11y". In principle this should run everything in tests/test_playwright.py, but at the top of that file there's an playwright = pytest.importorskip("playwright") - which I think means we are skipping all playwright tests, since playwright is not a dependency (except in the a11y-tests environment, which we aren't using here). Can someone confirm?

[gabalafou]

This test is great! exactly at the level I was hoping for. I left a few requests inline

[gabalafou]

I think CI is not running the playwright tests

That sounds right to me, based on what you wrote. I think that's probably why a number of tests like this have been snuck into the test_a11y file, even though they are not super specific to accessibility. You could take the same approach. I don't think you really need a test fixture for your test unless you were trying to be more fine grained about what you're testing. But if your test is just the following two parts:

1. Scroll to first heading, check that the first TOC entry is highlighted
2. Scroll to bottom of page, check that the first TOC entry is not highlighted

then I think you can reuse one of the docs page built for test_a11y

[drammock]

So CI runs for example the py312-sphinx61-tests tox command, which invokes coverage run -m pytest -m "not a11y". In principle this should run everything in tests/test_playwright.py, but at the top of that file there's an playwright = pytest.importorskip("playwright") - which I think means we are skipping all playwright tests, since playwright is not a dependency (except in the a11y-tests environment, which we aren't using here). Can someone confirm?

🤦🏻 sorry this was probably my oversight in #1861

[peytondmurray]

Some tests were failing. After looking more closely, I noticed that we were running e.g. tox run -e compile-assets,i18n-compile,py39-sphinx61-tests. I could be mistaken, but it seems like there is no point in running compile-assets and i18n-compile if we are not also building the docs with e.g. py39-docs.

I just added py39-docs to CI.yml so that the version switcher test runs correctly.

[peytondmurray]

With #2133 merged, this should be ready to go. @drammock @gabalafou @trallard as long as this looks okay to you all, I think we can merge.

[trallard]

I will leave this to @gabalafou to review.
Also if this is ready @peytondmurray could you remove WIP from the PR to signal its status?

[peytondmurray]

Thanks for converting the TOC test to use a separate test page. I tested the new test locally and it runs successfully. Are we good to merge here?

[peytondmurray]

This explanation will be really useful to anyone that reads through next. Thanks for this!

[drammock]

wheee, in it goes! thanks @peytondmurray

[12rambau]

I didn't followed-up the whole dev process but when checking this page on latest: https://pydata-sphinx-theme.readthedocs.io/en/latest/examples/kitchen-sink/api.html#urllib.parse.unquote_plus the scrollspy still fail at catching the anchors with "." is it normal ?

[drammock]

I didn't followed-up the whole dev process but when checking this page on latest: https://pydata-sphinx-theme.readthedocs.io/en/latest/examples/kitchen-sink/api.html#urllib.parse.unquote_plus the scrollspy still fail at catching the anchors with "." is it normal ?

argh. I didn't test carefully before merging. On this page if I click the anchor link for "Epigraph" the scrollspy highlights "Pull Quotes":