Multimodal: vision language models (VLMs)

Definition

Image-text-to-text models, also known as vision language models (VLMs), are language models that take an image input. These models can tackle various tasks, from visual question answering to image segmentation. This task shares many similarities with image-to-text, but with some overlapping use cases like image captioning.

Image-to-text models only take image inputs and often accomplish a specific task, whereas VLMs take open-ended text and image inputs and are more generalist models. Transformers' vision language model guide provides a brief overview of VLMs and shows how to use them with Transformers for inference.

At a glance

There are multiple types of VLMs. Transformers supports many model quantization libraries, and the vision language model guide only shows int8 quantization with Quanto.

Base models are used for fine-tuning.
Chat fine-tuned models are for conversation.
Instruction fine-tuned models.
With Quanto, to quantize a model during loading, first create a QuantoConfig.
Then load the model as usual, but pass the quantization configuration during model initialization.
8-bit is not a CUDA-native precision, so the weights are quantized back and forth on the fly, which adds up to latency.

The source does not state the order of these points, so they are listed without arrows.

How it works

The vision language model guide's example model, Qwen/Qwen3-VL-4B-Instruct, has a chat template that helps parse chat outputs. The example model can also accept multiple images as input in a single conversation or message. Alternating between the user and assistant role grounds the model with prior context to generate better responses.

Transformers supports streaming with the TextStreamer or TextIteratorStreamer classes, and text streaming is used for a better generation experience. Handling generation in a separate thread allows the generated text tokens to be streamed in real-time. VLMs are often large and need to be optimized to fit on smaller hardware.

Like text-only models, multimodal models track dialogue history with a chat template. If the template is applied with processing, the entire conversation gets reprocessed every turn because Jinja cannot distinguish previously encoded inputs from new ones. Caching lets you reuse the encoded representations of images or audio passed across turns instead of reprocessing them each time, cutting redundant computation.

Example

The vision language model guide focuses on inference with an instruction-tuned model. With the Pipeline API, you specify the image-text-to-text task and the model you want to use. You can also load the images separately and pass them to the pipeline.

The images are still included in the input text field of the output. The streaming example preprocesses the inputs as usual and initializes TextIteratorStreamer to handle the generation in a separate thread. Dynamic-resolution models like Qwen3-VL and Qwen3.5 default to a ~16MP longest edge, which can emit 10,000+ visual tokens per image and stall inference or run out of memory on consumer GPUs.

Setting the minimum and maximum pixel values when loading the processor caps this. The image-text-to-text task page covers model types, use cases, datasets, and more.

Related tools listed on this site.