VBench: Benchmark Suite for Video Generative Models

Core facts

Read by a script from the official API, not rewritten by a model.

Repository: https://github.com/Vchitect/VBench
License: Apache-2.0
Languages: Python
Maintainer: Vchitect
Homepage: https://vchitect.github.io/VBench-project/
Verified: 2026-10-31

What it is

This repository provides unified implementations for the VBench series of works, which the project says support comprehensive evaluation of video generative models across a wide spectrum of capabilities and settings. The authors propose VBench, a comprehensive benchmark suite for video generative models. VBench++ extends VBench with VBench-I2V for image-to-video, VBench-Long for long videos, and VBench-Trustworthiness covering fairness, bias, and safety. VBench-2.0 extends VBench to evaluate intrinsic faithfulness.

Main features

The VBench project says it designed a comprehensive and hierarchical Evaluation Dimension Suite that decomposes "video generation quality" into multiple well-defined dimensions to facilitate what it calls fine-grained and objective evaluation.
The VBench project says that, for each evaluation dimension, its Evaluation Method Suite uses carefully crafted methods or a designated pipeline for automatic objective evaluation.
VBench++ evaluates not only technical quality but also the trustworthiness of generative models.
VBench-2.0 introduces a structured evaluation suite comprising five broad categories and 18 fine-grained capability dimensions.

Getting started

Use VBench to evaluate videos and video generative models.
To evaluate your own videos with VBench, simply provide the path to the video file, or the path to the folder that contains your videos.
When evaluating your own videos with VBench, there is no requirement on the videos' names.
Detectron2 works only with CUDA 12.1 or 11.X.
Before evaluating the temporal flickering dimension, it is necessary to filter out the static videos first.

From the README

pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118 # or any other PyTorch version with CUDA<=12.1
pip install vbench

From the README

   pip install detectron2@git+https://github.com/facebookresearch/detectron2.git

From the README

python evaluate.py \
    --dimension $DIMENSION \
    --videos_path /path/to/folder_or_video/ \
    --mode=custom_input

From the README · bash

vbench evaluate --videos_path $VIDEO_PATH --dimension $DIMENSION

From the README · python

from vbench import VBench
my_VBench = VBench(device, <path/to/VBench_full_info.json>, <path/to/save/dir>)
my_VBench.evaluate(
    videos_path = <video_path>,
    name = <name>,
    dimension_list = [<dimension>, <dimension>, ...],
)

Who it is for

VBench is designed for evaluating different models on a standard benchmark. The authors provide scripts for calculating the Total Score, Quality Score, and Semantic Score in the Leaderboard. The Total Score is a weighted average of the Quality Score and Semantic Score.

In this calculation, each dimension's results are normalized using a formula. The VBench repository has released evaluation code for 16 text-to-video evaluation dimensions. VBench-I2V supports evaluating image-to-video models.

VBench-Trustworthiness supports evaluating the trustworthiness of video generative models, such as culture, fairness, bias, and safety. While VBench primarily assessed early-stage technical quality, VBench-2.0 expands the benchmarking framework to evaluate more advanced capabilities. VBench-2.0 supports evaluating customized videos.

The authors release a pipeline for evaluating human anatomical quality in AIGC videos. The authors release all the videos sampled and used for VBench evaluation. The authors present radar charts separately for the evaluation results of open-source and closed-source models.

The VBench-Long leaderboard now has 10 long video generation models. The VBench leaderboard now has 40 text-to-video models, both long and short. VBench++ was accepted to TPAMI.

Related tools listed on this site.