Refactor LLM chats: separate streaming logic and enforce strict typing

[s-alexey]

Major refactoring of the LLM interaction layer, significantly enhancing the llm.prompt method to establish it as the primary, unified entry point for all model communications. The goal is to abstract away model-specific logic, providing seamless support for structured outputs, automatic tool calling, and vision capabilities across all integrated models.

This simplifies task definitions and enhances the user experience by providing a consistent, high-level API.

• Enhanced llm.prompt with automatic tool calling:

  • The llm.prompt method has been upgraded to manage the entire conversation turn, now including a built-in, multi-step tool-calling loop.
  • When tools are provided, prompt automatically orchestrates the interaction: it invokes the LLM, executes requested tools, sends the results back, and repeats this cycle until a final answer is generated. This eliminates the need for manual tool-handling logic in task definitions.

• Automatic tool-calling emulation:

  • For models that lack native tool-calling support, a new emulation layer transparently provides this functionality by wrapping the requests with structured prompts, making the feature available across all models.

• Refactored Actor model:

  • The llms.py module has been streamlined. API-specific logic has been moved into dedicated actors/genai.py and actors/openai.py modules.
  • Experimental streaming functionality is now encapsulated in separate classes (e.g., StreamingGoogleGenAI, StreamingOpenAIResponsesAPI) to isolate it from the core API used for scheduled runs

• Improved vision and image support:

• Enhanced support for multimodal inputs, particularly for the Gemini API. The framework now correctly handles image content, including captions and various data formats (URLs and base64).

• New agentic assertion:

• Added assert_tool_was_invoked to allow for testing and evaluation of agentic behavior by verifying that a specific tool was used during a task.

• Updated Examples & Tests

  • Revised examples to demonstrate the simplified llm.prompt API for tool use.
  • Added a comprehensive suite of API integration tests (test_api_integration.py) that run against live OpenAI, Google, and Model Proxy endpoints (when API keys are available) to ensure cross-model consistency.

[dolaameng]

Thanks for the refactoring, which is really helpful and clearer! Left some questions/comments to understand more.

We can keep iterating it.

[develra]

LGTM - it's a bit hard for me to tell as a person pretty ignorant of this code if the test coverage is sufficient to be confident that these changes are safe. I think that it would be good to think through what might break as a result of these changes and make sure we have test coverage for it - especially given the somewhat sensitive timing of a new launch.

[dolaameng]

qq: If this is None, will it throw Type Error in invoke_tool for functions without arguments? so we should use calls.pop(0).args or {} instead.

[dolaameng]

if support_tool_calling is True, would the tool be called twice, once by API native support, the other by our "simulated call"? For example, would this work?

# %%
# --- Test Case: Tool called twice ---

COUNTER = 0


def increment_counter() -> int:
    global COUNTER
    COUNTER += 1
    return COUNTER


@benchmark_test(include=[
  "google/gemini-2.5-pro",
])
@kbench.task()
def test_stateful_tool_double_execution(llm):
    global COUNTER
    COUNTER = 0  # Reset for each test run

    llm.prompt("Call the increment_counter tool.", tools=[increment_counter])

    # If the bug exists, this will fail because COUNTER will be 2 (or more).
    kbench.assertions.assert_equal(
        1, COUNTER, expectation="Tool should be executed exactly once."
    )

[dolaameng]

It seems the role system will NOT be recognized as "tool result" by some models like gemini. This will cause an infinite calling of tools till ToolInvocationLimitExhausted reached. Shall we change the role to "user"?

For example, this seems to fail the same test as in https://github.com/Kaggle/kaggle-benchmarks/pull/12/changes#r3018051338

[dolaameng]

Same line here: shall we match function_response to function_call by id (or name) instead of pop(0). See here it seems to be the way to associate function results to function calls.

This might result in misaligned results (I remember seeing it before):

function_call(add, 2, 3)
function_call(times, 4, 5)

function_response(times, 20) # this will be mis-assigned to add
function_response(add, 5)