中文

VLMEvalKit: evaluation toolkit for large vision-language models

VLMEvalKit is an open-source evaluation toolkit of large vision-language models (LVLMs); its Python package name is vlmeval.

Verified

Core facts

Read by a script from the official API, not rewritten by a model.

License
Apache-2.0
Languages
Python
Maintainer
open-compass
Verified

What it is

VLMEvalKit (the Python package name is vlmeval) is an open-source evaluation toolkit of large vision-language models (LVLMs). VLMEvalKit enables one-command evaluation of LVLMs on various benchmarks, without the heavy workload of data preparation under multiple repositories. In VLMEvalKit, generation-based evaluation is adopted for all LVLMs, and evaluation results are provided from both exact matching and LLM-based answer extraction. VLMEvalKit uses generation-based evaluation for all VLMs, optionally with LLM-based answer extraction.

Main features

  1. VLMEvalKit can now save prediction files in TSV format.

  2. VLMEvalKit supports multi-node distributed inference using LMDeploy (which supports InternVL Series, QwenVL Series and LLaMa4) or VLLM (which supports QwenVL Series and LLaMa4).

  3. The image and video benchmarks supported by VLMEvalKit number 70+.

  4. The LMMs supported by VLMEvalKit include commercial APIs, open-source models, and more (200+).

  5. The models supported by VLMEvalKit include InternVL3 Series, Gemini-2.5-Pro, Kimi-VL and LLaMA4, among others.

  6. The benchmarks supported by VLMEvalKit include MMMU-Pro, WeMath, 3DSRBench, LogicVista, VL-RewardBench, CC-OCR, CG-Bench, CMMMU and WorldSense.

Getting started

  1. For the Qwen3.5 series, the project recommends transformers 5.2.0 or later.

  2. For the Moondream series and Aria, the project recommends torchvision 0.16 or later.

  3. For Moondream1, the project recommends transformers 4.36.2.

  4. For AKI, the project recommends transformers 4.42.0.

  5. For Aria, the project recommends transformers 4.45.0.

  6. For Moondream2 and the H2OVL series, the project recommends transformers 4.44.0.

  7. For the Molmo series, the project recommends transformers 4.50.3, or alternatively 4.46.1, 4.51 or 4.53.

From the README · python
# Demo
from vlmeval.config import supported_VLM
model = supported_VLM['idefics_9b_instruct']()
# Forward Single Image
ret = model.generate(['assets/apple.jpg', 'What is in this image?'])
print(ret)  # The image features a red apple with a leaf on it.
# Forward Multiple Images
ret = model.generate(['assets/apple.jpg', 'assets/apple.jpg', 'How many apples are there in the provided images? '])
print(ret)  # There are two apples in the provided images.

Who it is for

The performance numbers on the project's official multi-modal leaderboards can be downloaded. Some VLMs may not be able to run under certain transformer versions. The VLMEvalKit codebase is not designed to reproduce the exact accuracy number reported in the original papers of all third-party benchmarks.

Some benchmarks may use different approaches, for example SEEDBench uses PPL-based evaluation. For benchmarks that use different approaches, VLMEvalKit compares both scores in the corresponding result. By default, VLMEvalKit uses the same prompt template for all VLMs to evaluate on a benchmark.

Some VLMs may have their own specific prompt templates, and some of these are not covered by the codebase at this time. The benchmarks supported by VLMEvalKit also include HLE-Bench, MMVP and MM-AlignBench, among others.

Sources

This page is written from the official sources below. Open them to check the original text.

Verified ·

This page is written by AI from the official sources above and checked automatically. This site has not tested it. Found a mistake? Tell us.