Reimplement CancellableQueryTracker

[reubent]

I don't have access to JIRA to create a ticket but please see below. I've posted in the users list but not received any replies to I've attempted to fix myself.

Description

We noticed increasing memory usage over long periods of time in our Solr Clouds. Running some synthetic tests against such showed that it was directly linked to the number of abnormally terminated requests received - either exceeding resource limits or client timeout closed connection.

Investigation showed that the length of the response received to the tasks/list endpoints was growing. The resource leak was minimal but the time spent processing them was rising therefore a cleanup mechanism was needed.

Further investigation of why they were not being removed showed that when code ran out of time to run the calls to checkLimitsBefore would cause them to return before the finaliser that removes the active query from the list runs. This issue only appears to affect

1. Explicitly set query IDs
2. Queries sent to shards which get the node name prepended to the generated query ID when adding to the list but not when being removed

Solution

1. Prevent CancellableQueryTracker from filling with abnormally terminated requests by adding a timeout mechanism and making it configurable
2. Improve logging in that class
3. Add tests for that class
4. Add an additional release for activeQueries when the request has been finished early

Additionally there are clearly issues with the cancellation mechanism more generally that I'm not going to tackle now but am more than happy to look at in more depth later. Namely:

1. Multithreaded queries are not cancellable
2. Cancellability is dependent on the cancellation being sent to the same node that created the task to be cancelled. It also seems there's something weird with shard selection going on here too but I've not got to the bottom of it.
3. activeTaskList is again node dependent so the same UUID can be simultaneously sent to two nodes

Tests

I've added tests for the class and run nearly half a million "broken" and half a million non broken requests against a cloud of four nodes

Checklist

Please review the following and check all that apply:

• I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
• I have created a Jira issue and added the issue ID to my pull request title.
• I have given Solr maintainers access to contribute to my PR branch. (optional but recommended, not available for branches on forks living under an organisation)
• I have developed this patch against the main branch.
• I have run ./gradlew check.
• I have added tests for my changes.
• [n/a] I have added documentation for the Reference Guide