mistralai/Mixtral-8x7B-Instruct-v0.1 fails in Docling RAG example with HuggingFaceEndpoint: Task mismatch (text-generation vs conversational)

[Kalindu-C]

Bug

Following the Docling documentation example for RAG pipelines, replacing the LLM with mistralai/Mixtral-8x7B-Instruct-v0.1 results in:

ValueError: Model mistralai/Mixtral-8x7B-Instruct-v0.1 is not supported for task text-generation and provider together. Supported task: conversational.

The documentation shows usage with task="text-generation", but this model is only supported for conversational.
Even when changing task="conversational", the same error occurs, which makes it unclear how to integrate conversational-only models with the example workflow.

Steps to reproduce

from pathlib import Path
from tempfile import mkdtemp
from dotenv import load_dotenv
import os
from langchain_core.prompts import PromptTemplate
from langchain_docling.loader import ExportType
from docling.chunking import HybridChunker
from langchain_docling import DoclingLoader
from langchain.chains import create_retrieval_chain
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain_huggingface import HuggingFaceEndpoint

def _get_env_from_colab_or_os(key):
    try:
        from google.colab import userdata
        try:
            return userdata.get(key)
        except userdata.SecretNotFoundError:
            pass
    except ImportError:
        pass
    return os.getenv(key)

load_dotenv()

HF_TOKEN = _get_env_from_colab_or_os("HF_TOKEN")
FILE_PATH = ["https://arxiv.org/pdf/2408.09869"]
EMBED_MODEL_ID = "sentence-transformers/all-MiniLM-L6-v2"
GEN_MODEL_ID = "mistralai/Mixtral-8x7B-Instruct-v0.1"
EXPORT_TYPE = ExportType.DOC_CHUNKS
QUESTION = "Which are the main AI models in Docling?"
PROMPT = PromptTemplate.from_template(
    "Context information is below.\n---------------------\n{context}\n---------------------\nGiven the context information and not prior knowledge, answer the query.\nQuery: {input}\nAnswer:\n",
)
TOP_K = 3
MILVUS_URI = str(Path(mkdtemp()) / "docling.db")

loader = DoclingLoader(
    file_path=FILE_PATH,
    export_type=EXPORT_TYPE,
    chunker=HybridChunker(tokenizer=EMBED_MODEL_ID),
)

docs = loader.load()
retriever = vectorstore.as_retriever(search_kwargs={"k": TOP_K})

llm = HuggingFaceEndpoint(
    repo_id=GEN_MODEL_ID,
    huggingfacehub_api_token=HF_TOKEN,
    task="text-generation"   # Also tried task="conversational"
)

question_answer_chain = create_stuff_documents_chain(llm, PROMPT)
rag_chain = create_retrieval_chain(retriever, question_answer_chain)

resp_dict = rag_chain.invoke({"input": QUESTION})
print(resp_dict["answer"])

Error:
ValueError: Model mistralai/Mixtral-8x7B-Instruct-v0.1 is not supported for task text-generation and provider together. Supported task: conversational.

What I Tried:

• Changing task="text-generation" → task="conversational".

• Confirmed that Mixtral-8x7B-Instruct-v0.1 works in HuggingFace directly with conversational/chat APIs.

• Using OpenAI model (gpt-4o-mini) with the same Docling example works fine.

• The error still occurs when switching to task="conversational" in Docling's HuggingFaceEndpoint.

Docling version

v2.44.0

Python version

Python: 3.11

[dosubot]

Hi @Kalindu-C! I'm Dosu and I’m helping the docling team.

The error happens because Docling relies on the upstream langchain_huggingface package to enforce model-task compatibility for HuggingFaceEndpoint, and Mixtral-8x7B-Instruct-v0.1 only supports the "conversational" task, not "text-generation"—but the RAG pipeline expects a text-generation-compatible LLM. Even when you set task="conversational", the workflow fails because the pipeline and HuggingFaceEndpoint expect outputs in the text-generation format, not the conversational format, and there are no Docling-side overrides or adapters for this behavior (source).

To use conversational-only models in this workflow, you would need to adapt the output format externally—such as by subclassing HuggingFaceEndpoint or postprocessing the conversational outputs to match what the RAG pipeline expects. Docling does not provide built-in support or documentation for this, so any workaround would need to be implemented outside Docling, and may require changes if the upstream libraries update their interfaces (source).

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[onestardao]

this is a classic no.13 – multi-agent chaos case from our 16-problem list: the model adapter, the provider adapter, and the pipeline task each expect a different schema.

what’s happening

• the docling example wires HuggingFaceEndpoint with task="text-generation".
• your provider stack reports the model as conversational-only, so the adapter refuses the text-generation task (and even flipping the flag still breaks because the payload shape doesn’t match what that provider expects).

quick ways to get it unstuck

1. pick one consistent stack:

   • HF Inference: use a model whose card explicitly supports text-generation and keep task="text-generation", or
   • Conversational path: set task="conversational" and send the provider’s expected chat payload (messages/history), not raw text.

2. if you’re actually routing through Together or another proxy, use their chat client (ChatTogether, ChatHuggingFace, etc.) instead of the plain HuggingFaceEndpoint, so the task ↔ payload mapping is handled for you.

3. minimal guardrails that catch this early:

   • assert (provider, task) compatibility on init (read from the model card / adapter map).
   • normalize to one internal schema, and transform at the edge per provider.

4. quick workaround:

   • swap to a strictly text-generation model on HF (e.g., a non-chat checkpoint) or
   • keep Mixtral-Instruct but switch to the chat client and pass chat messages.

we’ve helped 80+ devs fix similar cross-adapter mismatches; project is mit-licensed, cold start → ~600 stars in ~60 days. if you want the tiny “task-compat guard + schema normalizer” snippet we use, say the word and i’ll share.