inoculate test run timeout is bad and i should feel bad about it

[hawkw]

as I said in #511 (comment)

We should probably make the timeout much smarter than just "kill the
QEMU process after 20 minutes", and instead do it only if we don't see
any interesting test output, or something. But, this is better than just
always failing test runs because we're stupid about timeouts, I guess.

basically, cargo inoculate test has a --timeout-secs flag, which fails the test run if tests don't complete within a specified timeout. this is intended to fail the run due to issues like the kernel deadlocking or tests not running at all. however, the way we do this currently is Real Dumb and Bad: we spawn the QEMU process with spawn_timeout, and abruptly kill it if it hassn't already exited after the timeout. this means that we might still be legitimately running tests, and as we add more tests, the likelihood of that happening increases.

instead, we probably should do something where we reset the timeout every time a test completes successfully, so we only kill the VM if one test has been wedged for a really long time, rather than bounding the total length of test suite execution.

also, we probably should tell the user that the tests timed out, instead of just killing the VM and then getting confused because QEMU exited abruptly. right now, the dev experience is pretty bad: we kill the VM and then the test runner prints a bunch of stuff about how it was really surprised that QEMU just randomly exited for no reason. my brother in Christ, you were the one that killed him! this incorrectly makes the user think that the kernel has done something so evil and wrong that made the VM crash, instead of "oh your tests took slightly too long to run and we timed out".

this is probably best done by making the inoculate test runner bits async and using tokio::process::Command rather than std::process::Command, so that we can select over a test starting/finishing or a timeout firing, and stuff.