Streaming chat crashes with IndexError on the final chunk (empty choices) - Python SDK

[dipeshpandit12]

Streaming chat crashes with IndexError on the final chunk (empty choices) — Python SDK

Summary

When consuming complete_streaming_chat(...) from the Python SDK, the final stream chunk
carries an empty choices list. The SDK's own documented usage example indexes
chunk.choices[0] unconditionally, so any consumer following the canonical pattern crashes with
IndexError: list index out of range after the model has already produced its full answer.

• Component: foundry-local-sdk (Python)
• Severity: High (canonical usage crashes; the example in the code itself is affected)
• Type: Correctness / API-contract
• Status: Reproduced

Environment

SDK	foundry-local-sdk 1.2.3 (PyPI)
Python	3.13.5
OS / HW	macOS (Darwin 25.5.0), Apple silicon (arm64)
Execution provider	WebGpuExecutionProvider
Model	qwen2.5-0.5b (qwen2.5-0.5b-instruct-generic-gpu-4, v4)

Where the bug lives

sdk/python/src/openai/chat_client.py

• _on_chunk (~line 234) correctly iterates raw.get("choices", []) and can yield a
  ChatCompletionChunk whose choices list is empty (the terminating chunk).
• The docstring/example for complete_streaming_chat (~lines 273–274) demonstrates:

  for chunk in client.complete_streaming_chat(messages):
      if chunk.choices[0].delta.content:   # <-- IndexError on the empty final chunk
          ...

The library does not filter the empty chunk and does not document that it occurs, so it pushes the
failure onto every caller — including its own example, which the sample apps copy.

Steps to reproduce

1. Install the SDK and load a small model:

   python3 -m venv .venv && source .venv/bin/activate
   pip install foundry-local-sdk

2. Run the shipped sample samples/python/tutorial-chat-assistant/src/app.py (its streaming loop
   uses chunk.choices[0].delta.content), or the minimal repro below.

Minimal reproduction

from foundry_local_sdk import Configuration, FoundryLocalManager

FoundryLocalManager.initialize(Configuration(app_name="repro"))
manager = FoundryLocalManager.instance
model = manager.catalog.get_model("qwen2.5-0.5b")
model.download(); model.load()
client = model.get_chat_client()

messages = [{"role": "user", "content": "In one sentence, what is the golden ratio?"}]
for chunk in client.complete_streaming_chat(messages):
    print(chunk.choices[0].delta.content or "", end="", flush=True)  # crashes on final chunk
model.unload()

Actual behavior

The answer streams fully, then the loop crashes on the terminating chunk:

Assistant: The golden ratio is a mathematical constant ... at its optimal form.
Traceback (most recent call last):
  File ".../src/app.py", line 66, in main
    content = chunk.choices[0].delta.content
              ~~~~~~~~~~~~~^^^
IndexError: list index out of range

Expected behavior

Streaming completes cleanly with no exception. Empty-choices chunks should either be filtered by
the SDK or clearly documented, and the example must not index [0] unconditionally.

Impact

• The shipped tutorial samples/python/tutorial-chat-assistant/src/app.py crashes on the first
  turn — the crash happens after a correct answer, so it looks like a runtime failure to new users.
• Any application copying the documented streaming pattern is affected.
• The bug is timing-independent and reproduces on every streaming call.

Suggested fix

Guard the index everywhere a chunk is consumed, and fix the in-code example:

for chunk in client.complete_streaming_chat(messages):
    if not chunk.choices:            # final chunk can carry no choices
        continue
    content = chunk.choices[0].delta.content
    if content:
        print(content, end="", flush=True)

Options for the SDK maintainers (any one resolves it):

1. Update the complete_streaming_chat docstring/example to guard chunk.choices.
2. Have the SDK skip yielding choice-less chunks (or expose them only when explicitly requested).
3. Document that a terminating empty-choices chunk is emitted (OpenAI-compatible behavior).

Affected files to patch

• sdk/python/src/openai/chat_client.py (docstring/example, ~lines 273–274)
• samples/python/tutorial-chat-assistant/src/app.py (streaming loop, ~line 66)
• Any other Python samples that stream (tutorial-document-summarizer, native-chat-completions,
  tutorial-tool-calling) should be checked for the same choices[0] pattern.

Notes

Related audit surfaced adjacent robustness gaps in the same file worth tracking separately
(early-break thread/queue leak in the streaming generator; json.loads(response.data) with
data=None raising an unwrapped TypeError). Those are distinct from this crash.

[prathikr]

@dipeshpandit12 please open your PR directly against our repo (instead of your fork) and I can approve/merge it once CI tests pass. Thank you for your contribution!