send email alerts for unexpected RPC errors

[dan-da]

Logs show we have been experiencing some times when neptune-explorer calls neptune-core RPCs that timeout -- and no email alert is sent to admin.

neptune-explorer has a watchdog task that periodically calls the neptune-core network() RPC. If result is different than previous result (state-change) then a notification email is sent (service down or service restored). So this is pretty good at (a) detecting if neptune-core dies or network connection is lost/restored. However note that the network() RPC does not obtain a GlobalState lock, and thus it is mostly guaranteed to return fast, while RPC calls that do obtain the lock may exceed the deadline (without triggering any email alert).

I will need to think about how to alert for such a condition. The simplest is to send an alert for any request that fails, but this has obvious flaw that if explorer is gettng hundreds of http requests, we may be sending hundreds of alerts for the same problem. So something a bit more clever is required.

neptune-explorer log example:

2024-12-15T02:19:19.342703Z ERROR tarpc::client::in_flight_requests: DeadlineExceeded
2024-12-15T07:16:56.017601Z ERROR tarpc::client::in_flight_requests: DeadlineExceeded
2024-12-15T09:25:59.058451Z ERROR tarpc::client::in_flight_requests: DeadlineExceeded
2024-12-15T11:05:58.598625Z ERROR tarpc::client::in_flight_requests: DeadlineExceeded
2024-12-15T11:07:56.687158Z ERROR tarpc::client::in_flight_requests: DeadlineExceeded
2024-12-15T15:15:45.362075Z ERROR tarpc::client::in_flight_requests: DeadlineExceeded
2024-12-15T22:29:11.566064Z ERROR tarpc::client::in_flight_requests: DeadlineExceeded
2024-12-15T22:52:00.371088Z ERROR tarpc::client::in_flight_requests: DeadlineExceeded
2024-12-15T23:50:00.141787Z ERROR tarpc::client::in_flight_requests: DeadlineExceeded
2024-12-16T03:03:27.506728Z ERROR tarpc::client::in_flight_requests: DeadlineExceeded
2024-12-16T03:14:08.821469Z ERROR tarpc::client::in_flight_requests: DeadlineExceeded
2024-12-16T09:11:36.309482Z ERROR tarpc::client::in_flight_requests: DeadlineExceeded
2024-12-16T09:35:04.449024Z ERROR tarpc::client::in_flight_requests: DeadlineExceeded
2024-12-16T11:49:44.791816Z ERROR tarpc::client::in_flight_requests: DeadlineExceeded
2024-12-16T11:55:45.255944Z ERROR tarpc::client::in_flight_requests: DeadlineExceeded
2024-12-16T11:55:48.129722Z ERROR tarpc::client::in_flight_requests: DeadlineExceeded
2024-12-16T11:55:55.431168Z ERROR tarpc::client::in_flight_requests: DeadlineExceeded

[dan-da]

note: the neptune-core instance has been restarted with --max-peers 4. It's been reported this cures some RPC timeout problems, so will monitor the logs for a few days and see what happens.