Story: Batch asynchronous manifest generation blocks GET requests

[markpbaggett]

Overview of Problem

Last week, we had to change an environmental variable in Rancher to get irIIIFService to work correctly with DSpace 7. Doing this required a restart and caused Redis to lose its cache since it's all stored in virtual memory. As a result, I had to regenerate all our manifests. During regeneration, I discovered that if you have 10000 cached manifests but have to generate a lot more simultaneously (500), get requests for those cached manifests get blocked. This is because Spring times out before the request for the cached item is even made.

Reproducing

Reproducing this should not be done on production for obvious reasons but is relatively easy to see.

1. Simply restart irIIIFService on dev or pre. This should nuke all manifests.
2. Generate a manifest by visiting that URL directly. This can be any manifests so simple ones are fine.
3. It's probably useful to also open Rancher and monitor the irIIIFService logs to check that the manifest is indeed being pulled from cache.
4. Now, asynchronously request ~500 or so manifests and wait 30 seconds or so.
5. Now, open a new tab or cURL for your manifest again. You'll see you get a Spring timeout before your get request even appears in the irIIIIFService log.

[markpbaggett]

Here is an example. This afternoon, irIIIFService all of a sudden stopped send responses. Instead, Spring responded with a 501 Timeout. I opened shell and date && tail -f ir-iiif-service.log | grep "169475" to target the log for when irIIIFService finally saw the request for this cached manifest.

As you can see, it was nearly 7 minutes until it saw the request for this resource.