JMS consumer thread dies silently when a broker restarts, and the pod stays Ready #2031

Summary

ConsumerContextJms treats a broker-side session loss as a local shutdown and permanently kills
the consumer thread, while the process stays up looking healthy. A service in this state consumes
nothing, forever, and nothing notices.

This took dev's submit service out for 6h50m on 2026-08-24 and was only found by comparing log
volume between namespaces.

The code

vcell-server/src/main/java/cbit/vcell/message/jms/ConsumerContextJms.java:77

} catch (JMSException e) {
    if (!bProcessing || e instanceof javax.jms.IllegalStateException){
        // close() unblocks a thread parked in receive(); that is shutdown, not a
        // failure. Logging it as one and looping would spin on the closed consumer.
        lg.debug(toString()+" consumer closed while polling", e);
        break;
    }
    onException(e);
}

The intent is right: close() unblocks a thread parked in receive(), and treating that as a
failure would spin. The problem is the discriminator. javax.jms.IllegalStateException is also
what the provider throws when the broker recycles the session underneath a live consumer — which
is not shutdown, it is exactly the case that needs reconnection. break exits the poll loop and the
thread dies for the lifetime of the process.

bProcessing alone already distinguishes deliberate shutdown; it is the instanceof clause that
overreaches.

What happened

A network/storage outage at ~19:50Z restarted dev's ActiveMQ brokers. Then:

19:59:16.960Z  ConsumerContextJms@6796aa72 consumer thread exiting.

...and that is the last line submit ever logged. Over the following six hours:

The pod stayed Running with 0 restarts. sched kept dispatching a health-check simulation
every 5 minutes and publishing MessageType='SimulationJob'; nothing consumed it; each attempt
timed out after 8 minutes:

dev  Critical  "simulation failed (892): simulation took longer than 480000 to complete"  elapsed 481,280 ms
prod OK                                                                                   elapsed   4,645 ms

JmsFailoverWatchdog did not help, and could not: it guards the connection, not consumer
threads. That is why sched recovered (the watchdog exited the JVM and the pod restarted) and
submit did not.

Not a regression from a recent release

Introduced in 6fabc1f871 (2026-08-06) and present from 8.0.10.01 onward — so both prod
(8.0.26.01) and dev (8.1.1.01) contain it. Prod is unaffected only because its brokers have not
restarted in 65 days; dev's restarted during the outage. Any broker restart in prod would produce
the same silent failure.

Suggested fix

Distinguish "we are shutting down" from "the broker went away", rather than inferring shutdown from
the exception type:

• gate the break on bProcessing alone, and treat IllegalStateException while still processing
  as a reconnect condition;
• or have the consumer re-establish its session rather than exit;
• and, if the thread does exit while bProcessing is true, make that loud — it currently logs at
  info on the way out and debug for the cause, which is why six hours passed unnoticed.

Worth deciding alongside #2032 (no livenessProbe on these services), which would have caught this from the outside.

Reproducing

Restart an ActiveMQ broker under a running submit/sched/data/db and watch for
consumer thread exiting. followed by silence, with the pod still Ready.