PDF fails with Qwen-3 VL 235B A22B and others

[leonardmq]

Wanted to set up Qwen-3 VL 235B A22B as an extractor model, but:

1. The thinking variant does not work - the content is an empty string, and the actual content we expect is in reasoning_content instead, causing an exception.
2. LiteLLM validates the annotations coming through from the provider when the input is a PDF file, so even though the request succeeds, LiteLLM rejects the annotations from Qwen (it expects type=url_citation but the model returns type=file).

I ran into Problem 2 once before when trying to add multimodal input for Step3. Updating litellm to 1.77.3 does not fix it

Problem 2 stack trace:

Traceback (most recent call last):
  File "XXX/main.py", line 77, in <module>
    asyncio.run(main(file_path, file_type))
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "XXX/.local/share/uv/python/cpython-3.13.7-macos-aarch64-none/lib/python3.13/asyncio/runners.py", line 195, in run
    return runner.run(main)
           ~~~~~~~~~~^^^^^^
  File "XXX/.local/share/uv/python/cpython-3.13.7-macos-aarch64-none/lib/python3.13/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "XXX/.local/share/uv/python/cpython-3.13.7-macos-aarch64-none/lib/python3.13/asyncio/base_events.py", line 725, in run_until_complete
    return future.result()
           ~~~~~~~~~~~~~^^
  File "XXX/main.py", line 57, in main
    response = await litellm.acompletion(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<12 lines>...
        )
        ^
  File "XXX/.venv/lib/python3.13/site-packages/litellm/utils.py", line 1601, in wrapper_async
    raise e
  File "XXX/.venv/lib/python3.13/site-packages/litellm/utils.py", line 1452, in wrapper_async
    result = await original_function(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "XXX/.venv/lib/python3.13/site-packages/litellm/main.py", line 576, in acompletion
    raise exception_type(
          ~~~~~~~~~~~~~~^
        model=model,
        ^^^^^^^^^^^^
    ...<3 lines>...
        extra_kwargs=kwargs,
        ^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "XXX/.venv/lib/python3.13/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py", line 2278, in exception_type
    raise e  # it's already mapped
    ^^^^^^^
  File "XXX/.venv/lib/python3.13/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py", line 2216, in exception_type
    raise APIConnectionError(
    ...<7 lines>...
    )
litellm.exceptions.APIConnectionError: litellm.APIConnectionError: APIConnectionError: OpenrouterException - Invalid response object Traceback (most recent call last):
  File "XXX/.venv/lib/python3.13/site-packages/litellm/litellm_core_utils/llm_response_utils/convert_dict_to_response.py", line 527, in convert_to_model_response_object
    message = Message(
        content=content,
    ...<8 lines>...
        images=choice["message"].get("images", None),
    )
  File "XXX/.venv/lib/python3.13/site-packages/litellm/types/utils.py", line 643, in __init__
    super(Message, self).__init__(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        **init_values,  # type: ignore
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        **params,
        ^^^^^^^^^
    )
    ^
  File "XXX/.venv/lib/python3.13/site-packages/pydantic/main.py", line 253, in __init__
    validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
pydantic_core._pydantic_core.ValidationError: 1 validation error for Message
annotations.0.type
  Input should be 'url_citation' [type=literal_error, input_value='file', input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/literal_error

Minimal example to reproduce:

import asyncio
from pathlib import Path
from typing import Any, Literal
import litellm
import base64
import os

OPENROUTER_API_KEY = os.getenv("OPENROUTER_API_KEY")
if not OPENROUTER_API_KEY:
    raise ValueError("OPENROUTER_API_KEY is not set")


SupportedFileTypes = Literal["pdf", "image"]


def to_base64_url(mime_type: str, bytes: bytes) -> str:
    base64_url = f"data:{mime_type};base64,{base64.b64encode(bytes).decode('utf-8')}"
    return base64_url


def encode_file_litellm_format(path: Path, file_type: SupportedFileTypes) -> dict[str, Any]:
    match file_type:
        case "pdf": 
            return {
                "type": "file",
                "file": {
                    "file_data": to_base64_url("application/pdf", path.read_bytes()),
                },
            }
        case "image":
            return {
                "type": "image_url",
                "image_url": {
                    "url": to_base64_url("image/jpeg", path.read_bytes()),
                },
            }
        case _:
            raise ValueError(f"Unsupported file type: {file_type}")


async def main(file_path: Path, file_type: SupportedFileTypes):
    response = await litellm.acompletion(
            model="openrouter/qwen/qwen3-vl-235b-a22b-instruct",
            messages=[
                {
                    "role": "user",
                    "content": [
                        {"type": "text", "text": "What is this file about?"},
                        encode_file_litellm_format(
                            file_path, file_type
                        ),
                    ],
                }
            ],
        )
    return response


if __name__ == "__main__":
    # image: it works fine
    # file_type: SupportedFileTypes = "image"
    # file_path = Path("test.jpeg")

    # pdf: it throws on validation of the response
    file_type: SupportedFileTypes = "pdf"
    file_path = Path("test.pdf")
    
    asyncio.run(main(file_path, file_type))

Originally posted by @leonardmq in #646 (comment)

[leonardmq]

tldr:

• OpenRouter parses PDFs with MistralOCR when the model does not support PDFs
• It returns annotations to let the user reuse the output of MistralOCR in subsequent calls to the model
• LiteLLM does not support those annotations and raises during response validation

Opened an issue: BerriAI/litellm#15098