Make Modal polling more resilient

[WillEngler]

This PR helps us work around the backend memory leak problem when polling on a long-ish running Modal function.

After 30 seconds we'll wait for 5 seconds when polling. And if we see a 502 or 504, that usually means the server is restarting. We can just wait for a little and try again in that case.

---

📚 Documentation preview 📚: https://garden-ai--650.org.readthedocs.build/en/650/

[codecov]

Codecov Report

❌ Patch coverage is 94.11765% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 40.32%. Comparing base (0489a92) to head (067b64c).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
garden_ai/backend_client.py	94.11%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #650      +/-   ##
==========================================
+ Coverage   39.30%   40.32%   +1.02%     
==========================================
  Files          30       30              
  Lines        1743     1758      +15     
==========================================
+ Hits          685      709      +24     
+ Misses       1058     1049       -9     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[OwenPriceSkelly]

LGTM (with a small possible gotcha, nothing blocking)

[OwenPriceSkelly]

nit: I'm 95% sure that we want extra parens around ((e.response.status_code == 502 or e.response.status_code == 504) and retry_count <= 3)

[WillEngler]

I'll get that in!