PFT timeouts

[harrislapiroff]

We've seen a number of timeouts in our alerts recently. @maeve-fpf will investigate whether bumping up resources stops those.

[update 11/15] Next steps here are for the web team to investigate the performance of the export endpoint. (Personally I'd like to see the export endpoint and the API unified so we don't have either diverging features or performance. Export links are functionally basically API calls with a CSV output format.)

[chigby]

The last time I looked into this it was part of #1573

[chigby]

Example of a segment of logs leading up to the worker timeout event:

(screenshot in case it's easier to see).

[soleilera]

@maeve-fpf will continue investigation on resource allocation to improve performance. We also created a web team card to timebox a performance investigation on the web app side.

[maeve-fpf]

We've pushed memory request/limit increases for the Django pods, and will continue to monitor for another 2 weeks then close.

[soleilera]

It's been 2 weeks and I believe PFT timeouts are no longer an issue so I think we can close this, but defer to @maeve-fpf.

[harrislapiroff]

We're still seeing occasional timeouts, most recently on 11/10. It may be that the next step is on the web team to investigate performance of specific endpoints. I recall we have a conversation about this at web sync recently, does anyone remember the conclusion there? (attn @chigby @SaptakS)

[chigby]

I think we said we were going to investigate the "export" function?

[harrislapiroff]

That sounds right to me. Maybe the next steps here belong to the web team @soleilera. I'll bounce this to our hi pri list

[chigby]

We haven't seen timeouts in several weeks, closing in favor of #1802