[Bug] Pdf Input in Multimodal Models not possible?

[DaniloTrotta95]

Hello Guys,

I have been experimenting with the package, in one of our usecase we would like to use a multimodal model, where we give a base64 pdf file to the chat completion, with a output response schema. The goal is to let the Model fill out the schema using this pdf file.

Sadly I can't seem to get it to work.

const result = await llm.chat({ messages: [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": prompt
          },
          {
            "type": "file",
            "mimeType": "application/pdf",
            "data": pdfBase64
        },
        ]
      }
    ] })

This satisfies TS, but gets a 400 error: Error: 400 Missing required parameter: 'messages[0].content[1].file
(This is using a openai Model gpt-4o, using a Gemini Model for example gives a other Error, that the type file does not exist)

A other appraoch, like openRouter has it defined looks like this:

 {
            type: 'file',
            file: {
              filename: 'document.pdf',
              mimeType: 'application/pdf',
              file_data: pdfBase64,
            },
          },

I tried this appraoch aswell, but I get the Error: Only PDF files are supported!

It seems depending on the model the structure of how to pass the pdf to the model seems to be different. But even after trying multiple different approaches, I can't seem to get it to work.

Help would be appreciated!
Thanks!