OOM Crashes are still happening in the worker

[josephjclark]

We've tightened up some processing in the worker to make the main thread more resiliant.

But we're still seeing OOM explosions.

Here's one in GCP for 832eaf62-4bed-4c8b-bebe-6a0712e70aaa

Couple of notes:

• the last payload is about 5mb on disk pretty-printed. So not big I'd say.
• I think the worker blew up around 1gb of memory, but I'm not good at understanding the stack trace
• I don't think the worker is under particularly high load

I suspect this has more to do with the main worker thread processing many smaller events, than being blown up by one big event. But I'm really not sure.

Might be related to #897

Might be related to #826

[josephjclark]

Perhaps more than anything I'd really really like a way to notify us and the app about these OOM kills. But the nature of it means we can't do much: IIRC we do get a callback hook but it's strictly synchronous.