Scheduled jobs often fail due to db timeout

[ehuss]

The scheduled jobs thread regularly fails with the following:

database connection error: error communicating with the server: Connection timed out (os error 110)
thread 'main' panicked at /src/db.rs:239:47:
called `Result::unwrap()` on an `Err` value: Getting jobs data
Caused by:
    connection closed
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
2025-01-23T17:33:20.088253Z ERROR triagebot: run_scheduled_jobs task died (error=panic)

or

database connection error: error communicating with the server: Connection timed out (os error 110)
thread 'main' panicked at src/main.rs:377:26:
called `Result::unwrap()` on an `Err` value: database schedule jobs
Caused by:
    0: Inserting job
    1: connection closed
2025-01-23T17:37:11.484202Z ERROR triagebot: schedule_jobs task died (error=panic)

I'm uncertain what might be happening. There is code to check for closed connections, but maybe it is getting dropped without being closed properly? It seems peculiar that it is only affecting the scheduled jobs, since I would expect stale cached connections to affect anything touching the database.

[ehuss]

I would expect stale cached connections to affect anything touching the database.

My new theory is that it is affecting webhooks, however it is manifesting as a hang, and the webhook gets killed after 10s, and there is no log message when the webhook thread is killed.