Internal Server Error (500) when using Gemini API with Inspect framework #573

lennijusten · 2024-09-26T00:51:47Z

Description of the bug:

Description

I'm consistently encountering an Internal Server Error (HTTP 500) when trying to use the Google Gemini API through the Inspect evaluation framework. This error occurs during the generate_content call. I'm currently on the free trial.

Steps to Reproduce

Set up an evaluation using the Inspect framework
Configure the evaluation to use the Gemini model (in my case, google/gemini-1.5-pro)
Run the evaluation

Error Message

InternalServerError: 500 An internal error has occurred. Please retry or report in https://developers.generativeai.google/guide/troubleshooting

Environment

Operating System: MacOS Sonoma 14.5
Python version: 3.12.4

Package Versions

google-ai-generativelanguage: 0.6.6
google-api-core: 2.19.2
google-api-python-client: 2.143.0
google-auth: 2.34.0
google-auth-httplib2: 0.2.0
google-generativeai: 0.7.2
googleapis-common-protos: 1.65.0

Full error traceback:

╭─ benchmarks/gpqa (78 samples): google/gemini-1.5-pro ─────────────────────────────────────────────────────────────────────╮
│ ╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮   dataset: (samples) │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/inspect_ai/… │       scorer: choice │
│ │ in task_run                                                                                      │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/inspect_ai/… │                      │
│ │ in task_run_sample                                                                               │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/inspect_ai/… │                      │
│ │ in __call__                                                                                      │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/inspect_ai/… │                      │
│ │ in task_run_sample                                                                               │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/inspect_ai/… │                      │
│ │ in solve                                                                                         │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/inspect_ai/… │                      │
│ │ in generate                                                                                      │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/inspect_ai/… │                      │
│ │ in task_generate                                                                                 │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/inspect_ai/… │                      │
│ │ in generate                                                                                      │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/inspect_ai/… │                      │
│ │ in _generate                                                                                     │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/tenacity/as… │                      │
│ │ in async_wrapped                                                                                 │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/tenacity/as… │                      │
│ │ in __call__                                                                                      │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/tenacity/as… │                      │
│ │ in iter                                                                                          │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/tenacity/_u… │                      │
│ │ in inner                                                                                         │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/tenacity/__… │                      │
│ │ in <lambda>                                                                                      │                      │
│ │                                                                                                  │                      │
│ │ /opt/homebrew/Cellar/[email protected]/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.1… │                      │
│ │ in result                                                                                        │                      │
│ │                                                                                                  │                      │
│ │   446 │   │   │   │   if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]:                     │                      │
│ │   447 │   │   │   │   │   raise CancelledError()                                                 │                      │
│ │   448 │   │   │   │   elif self._state == FINISHED:                                              │                      │
│ │ ❱ 449 │   │   │   │   │   return self.__get_result()                                             │                      │
│ │   450 │   │   │   │                                                                              │                      │
│ │   451 │   │   │   │   self._condition.wait(timeout)                                              │                      │
│ │   452                                                                                            │                      │
│ │                                                                                                  │                      │
│ │ /opt/homebrew/Cellar/[email protected]/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.1… │                      │
│ │ in __get_result                                                                                  │                      │
│ │                                                                                                  │                      │
│ │   398 │   def __get_result(self):                                                                │                      │
│ │   399 │   │   if self._exception:                                                                │                      │
│ │   400 │   │   │   try:                                                                           │                      │
│ │ ❱ 401 │   │   │   │   raise self._exception                                                      │                      │
│ │   402 │   │   │   finally:                                                                       │                      │
│ │   403 │   │   │   │   # Break a reference cycle with the exception in self._exception            │                      │
│ │   404 │   │   │   │   self = None                                                                │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/tenacity/as… │                      │
│ │ in __call__                                                                                      │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/inspect_ai/… │                      │
│ │ in generate                                                                                      │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/inspect_ai/… │                      │
│ │ in generate                                                                                      │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/google/gene… │                      │
│ │ in generate_content_async                                                                        │                      │
│ │                                                                                                  │                      │
│ │   382 │   │   │   │   │   )                                                                      │                      │
│ │   383 │   │   │   │   return await generation_types.AsyncGenerateContentResponse.from_aiterato   │                      │
│ │   384 │   │   │   else:                                                                          │                      │
│ │ ❱ 385 │   │   │   │   response = await self._async_client.generate_content(                      │                      │
│ │   386 │   │   │   │   │   request,                                                               │                      │
│ │   387 │   │   │   │   │   **request_options,                                                     │                      │
│ │   388 │   │   │   │   )                                                                          │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/google/ai/g… │                      │
│ │ in generate_content                                                                              │                      │
│ │                                                                                                  │                      │
│ │    403 │   │   self._client._validate_universe_domain()                                          │                      │
│ │    404 │   │                                                                                     │                      │
│ │    405 │   │   # Send the request.                                                               │                      │
│ │ ❱  406 │   │   response = await rpc(                                                             │                      │
│ │    407 │   │   │   request,                                                                      │                      │
│ │    408 │   │   │   retry=retry,                                                                  │                      │
│ │    409 │   │   │   timeout=timeout,                                                              │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/google/api_… │                      │
│ │ in retry_wrapped_func                                                                            │                      │
│ │                                                                                                  │                      │
│ │   227 │   │   │   sleep_generator = exponential_sleep_generator(                                 │                      │
│ │   228 │   │   │   │   self._initial, self._maximum, multiplier=self._multiplier                  │                      │
│ │   229 │   │   │   )                                                                              │                      │
│ │ ❱ 230 │   │   │   return await retry_target(                                                     │                      │
│ │   231 │   │   │   │   functools.partial(func, *args, **kwargs),                                  │                      │
│ │   232 │   │   │   │   predicate=self._predicate,                                                 │                      │
│ │   233 │   │   │   │   sleep_generator=sleep_generator,                                           │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/google/api_… │                      │
│ │ in retry_target                                                                                  │                      │
│ │                                                                                                  │                      │
│ │   157 │   │   # This function explicitly must deal with broad exceptions.                        │                      │
│ │   158 │   │   except Exception as exc:                                                           │                      │
│ │   159 │   │   │   # defer to shared logic for handling errors                                    │                      │
│ │ ❱ 160 │   │   │   _retry_error_helper(                                                           │                      │
│ │   161 │   │   │   │   exc,                                                                       │                      │
│ │   162 │   │   │   │   deadline,                                                                  │                      │
│ │   163 │   │   │   │   sleep,                                                                     │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/google/api_… │                      │
│ │ in _retry_error_helper                                                                           │                      │
│ │                                                                                                  │                      │
│ │   209 │   │   │   RetryFailureReason.NON_RETRYABLE_ERROR,                                        │                      │
│ │   210 │   │   │   original_timeout,                                                              │                      │
│ │   211 │   │   )                                                                                  │                      │
│ │ ❱ 212 │   │   raise final_exc from source_exc                                                    │                      │
│ │   213 │   if on_error_fn is not None:                                                            │                      │
│ │   214 │   │   on_error_fn(exc)                                                                   │                      │
│ │   215 │   if deadline is not None and time.monotonic() + next_sleep > deadline:                  │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/google/api_… │                      │
│ │ in retry_target                                                                                  │                      │
│ │                                                                                                  │                      │
│ │   152 │                                                                                          │                      │
│ │   153 │   for sleep in sleep_generator:                                                          │                      │
│ │   154 │   │   try:                                                                               │                      │
│ │ ❱ 155 │   │   │   return await target()                                                          │                      │
│ │   156 │   │   # pylint: disable=broad-except                                                     │                      │
│ │   157 │   │   # This function explicitly must deal with broad exceptions.                        │                      │
│ │   158 │   │   except Exception as exc:                                                           │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/google/api_… │                      │
│ │ in __await__                                                                                     │                      │
│ │                                                                                                  │                      │
│ │    85 │   │   │   response = yield from self._call.__await__()                                   │                      │
│ │    86 │   │   │   return response                                                                │                      │
│ │    87 │   │   except grpc.RpcError as rpc_error:                                                 │                      │
│ │ ❱  88 │   │   │   raise exceptions.from_grpc_error(rpc_error) from rpc_error                     │                      │
│ │    89                                                                                            │                      │
│ │    90                                                                                            │                      │
│ │    91 class _WrappedStreamResponseMixin(Generic[P], _WrappedCall):                               │                      │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯                      │
│ InternalServerError: 500 An internal error has occurred. Please retry or report in                                        │
│ https://developers.generativeai.google/guide/troubleshooting

Actual vs expected behavior:

No response

Any other information you'd like to share?

No response

The text was updated successfully, but these errors were encountered:

gmKeshari · 2024-09-26T10:52:12Z

Hi @lennijusten ,

Have you been facing this issue since the beginning ? I mean is it like you were getting the response earlier and now facing this issue?

Try upgrading the google.generative library once and then run your commands :

pip install -q --upgrade google-generativeai

Please let us know if you are still having the issue.

lennijusten · 2024-09-26T20:37:45Z

Yes. It's been a semi-persistent problem throughout my usage of Gemini. Sometimes it works, but then after some number of requests or tokens, I get the Internal Server Error (500). The issue is also discussed here and here on Reddit.

I ran pip install -q --upgrade google-generativeai which upgraded me from google-generativeai==0.6.6 to google-generativeai==0.8.2 but the issue persists.

If there is some kind of rate limit happening it would be useful for the error code to reflect that.

With the current workings, I'm just burning tokens for nothing since my task never completes before the error.

gmKeshari · 2024-09-27T05:46:20Z

Hi @lennijusten . Sometimes we see Internal Server Error (500) because there are too many requests coming, which results in an unexpected error on Google's side. You can refer to this Troubleshooting doc "https://ai.google.dev/gemini-api/docs/troubleshooting?lang=python". I recommend temporarily switching to another model (e.g. from Gemini 1.5 Pro to Gemini 1.5 Flash) and see if it works.

Internal Server Error (500) is not related to rate limit, but free tier does have rate limits based model variation. Please refer to this doc : "https://ai.google.dev/gemini-api/docs/models/gemini" or "https://ai.google.dev/pricing"

If you want to upgrade to "Pay-as-you-go" tier, you need to set up a billing account.

MarkDaoust · 2025-02-19T14:30:23Z

Hi, thanks for the report.

There have been cases where bad inputs cause 500s. But it sounds like it wasn't consistent about which requests cause the error.

Like @gmKeshari said the service used to throw 500 errrors when it's overloaded (I thing they've changed that to a clearer error)

For intermittent errors, the SDK supports a 'retry' argument:

import google.generativeai as genai
from google.api_core import retry

model = genai.GenerativeModel('gemini-2.0-flash')

# For convenience, a simple wrapper to let the SDK handle error retries
def generate_with_retry(model, prompt):
  return model.generate_content(prompt, request_options={'retry':retry.Retry()})

I think the default limit is 5min, but it's configurable.

gmKeshari self-assigned this Sep 26, 2024

gmKeshari added status:triaged Issue/PR triaged to the corresponding sub-team component:python sdk Issue/PR related to Python SDK type:question Support-related issues labels Sep 26, 2024

lennijusten mentioned this issue Sep 27, 2024

Gemini - too many requests causing Internal Server Error (500) UKGovernmentBEIS/inspect_ai#545

Closed

MarkDaoust added component:api Issues related to the API, not the SDK. and removed component:python sdk Issue/PR related to Python SDK labels Feb 19, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Internal Server Error (500) when using Gemini API with Inspect framework #573

Internal Server Error (500) when using Gemini API with Inspect framework #573

lennijusten commented Sep 26, 2024 •

edited

Loading

gmKeshari commented Sep 26, 2024

lennijusten commented Sep 26, 2024 •

edited

Loading

gmKeshari commented Sep 27, 2024

MarkDaoust commented Feb 19, 2025

Internal Server Error (500) when using Gemini API with Inspect framework #573

Internal Server Error (500) when using Gemini API with Inspect framework #573

Comments

lennijusten commented Sep 26, 2024 • edited Loading

Description of the bug:

Description

Steps to Reproduce

Error Message

Environment

Package Versions

Actual vs expected behavior:

Any other information you'd like to share?

gmKeshari commented Sep 26, 2024

lennijusten commented Sep 26, 2024 • edited Loading

gmKeshari commented Sep 27, 2024

MarkDaoust commented Feb 19, 2025

lennijusten commented Sep 26, 2024 •

edited

Loading

lennijusten commented Sep 26, 2024 •

edited

Loading