[329] Implement alerting notifications on database_writer_pool failures

[Goodnessukaigwe]

Summary

• Track consecutive database_writer_pool write failures and emit a warning alert only when the configured threshold is first reached (WRITER_POOL_FAILURE_THRESHOLD, default 3).
• Detect stalled writes against WRITER_POOL_STALL_THRESHOLD_MS (default 120s) and report them once per quiet period, without incrementing the failure counter.
• Reset failure/stall tracking after a successful write so recovery requires a fresh consecutive run; alerts include operation name, counts, queue depth, and error message without SQL parameters or secrets.

Test plan

• Warnings trigger only after the configured consecutive error count
• Failures below the threshold do not emit alerts
• Successful writes clear the alert and require the full count again
• Stall alerts fire after the quiet window and are not duplicated
• Env thresholds are honored; invalid values fall back to defaults
• queueWrite integration covers failure, recovery, and stall paths

Closes #329

Made with Cursor

[drips-wave]

@Goodnessukaigwe Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits