langchain using with_structured_output via OpenAI client results in failure when using HF TGI 2.2.0?

[michael-newsrx]

I'm unclear as to the source of the issue I'm having. When I try and use langchain OpenAI client with with_structured_output and a schema, it throws an exception UnprocessableEntityError. Endpoint logs don't show anything I can find in all that [m mess as to why.

Here is my error:

I can't see what the API is actually returning in the exception object?

UnprocessableEntityError: Failed to deserialize the JSON body into the target type: response_format: missing field `value` at line 1 column 140

Here is my sheet:

def display(markdown: str, **kwargs) -> None:
    from IPython.core.display import display_markdown
    display_markdown(markdown, raw=True, **kwargs)
    

from huggingface_hub import InferenceEndpoint
from langchain.cache import SQLiteCache
from langchain.globals import set_llm_cache
from langchain_openai import ChatOpenAI

import local_utils
from config import use_llm_cache
from local_utils.hf.inference import hf_bearer_token
from notebooks_langchain_clarivate.config import llm_cache_file

bearer_token = hf_bearer_token()

if use_llm_cache:
    set_llm_cache(SQLiteCache(database_path=llm_cache_file))

ep: InferenceEndpoint = local_utils.hf.inference.llama_31_8B_Instruct(wait=True)
model = ChatOpenAI(model="TGI", base_url=ep.url + "/v1", api_key=bearer_token)

from pydantic.v1 import BaseModel, Field


class Joke(BaseModel):
    setup: str = Field(description="The setup of the joke")
    punchline: str = Field(description="The punchline to the joke")

structured_llm: ChatOpenAI
structured_llm = model.with_structured_output(Joke, method="json_mode")

import pathlib
from openai import UnprocessableEntityError

try:
    joke = structured_llm.invoke("Tell me a joke about cats.")
except UnprocessableEntityError as e:
    pathlib.Path("hf_request_with_error.json").write_text(e.request.content.decode("utf-8"))
    pathlib.Path("hf_response_with_error.json").write_text(e.response.content.decode("utf-8"))
    raise e

joke