VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with Video LLM

Yuqian Yuan1,2, Hang Zhang2, Wentong Li1, Zesen Cheng2, Boqiang Zhang2, Long Li1,2, Xin Li2, Deli Zhao2, Wenqiao Zhang1, Yueting Zhuang1, Jianke Zhu1, Lidong Bing2
1Zhejiang University 2DAMO Academy, Alibaba Group
arXiv Code Dataset(Comming soon)

📈

 Benchmark

🤗

 Model

🔥Highlights

In this work, we revisit the design of the Video LLM for finer-level video understanding. We contend that achieving this necessitates three essential components:

1. Dataset. Firstly, to achieve regional alignment between video content and language embeddings, we meticulously curate a large-scale object-text video instruction dataset named VideoRefer-700K.

2. Model. Next, we introduce an effective Video LLM, named VideoRefer, that enables fine-grained perceiving, reasoning and retrieval for user-defined regions at any specified timestamps. To accommodate both single-frame and multi-frame region inputs, we propose a versatile spatial-temporal object encoder.

3. Benchmark. Furthermore, to evaluate the regional video understanding capabilities of a Video LLM comprehensively, we develop a benchmark named VideoRefer-Bench, which consists of two sub-benchmarks: VideoRefer-Bench-D, which focuses on description generation from four aspects, and VideoRefer-Bench-Q, which emphasizes multiple-choice question answering across five aspects.

Abstract

Video Large Language Models (Video LLMs) have recently exhibited remarkable capabilities in general video understanding. However, they mainly focus on holistic comprehension and struggle with capturing fine-grained spatial and temporal details. Besides, the lack of high-quality object-level video instruction data and a comprehensive benchmark further hinders their advancements. To tackle these challenges, we introduce the VideoRefer Suite to empower Video LLM for finer-level spatial-temporal video understanding, i.e., enabling perception and reasoning on any objects throughout the video. Specially, we thoroughly develop VideoRefer Suite across three essential aspects: dataset, model, and benchmark. Firstly, we introduce a multi-agent data engine to meticulously curate a large- scale, high-quality object-level video instruction dataset, termed VideoRefer-700K. Next, we present the VideoRefer model, which equips a versatile spatial-temporal object encoder to capture precise regional and sequential representations. Finally, we meticulously create a VideoRefer-Bench to comprehensively assess the spatial-temporal understanding capability of a Video LLM, evaluating it across various aspects. Extensive experiments and analyses demonstrate that our VideoRefer model not only achieves promising performance on video referring benchmarks but also facilitates 025 general video understanding capabilities.

VideoRefer-700K Dataset

Interpolate start reference image.

We develop an automatic multi-agent data engine to create VideoRefer-700K, a large-scale and high-quality object-level video instruction-following dataset.

Visual samples from our VideoRefer-700 dataset

VideoRefer Model

Interpolate start reference image.

VideoRefer adopts a visual encoder and STC connector to encode the global scene-level visual representations, a pretrained text tokenizer to capture the language embeddings, and an instruction-following LLM for language decoding. To achieve video referring, we present a versatile and unified spatial-temporal encoder to derive object-level representations.

VideoRefer-Bench

VideoRefer-Bench assesses the models in two key areas: Description Generation, corresponding to VideoRefer-BenchD, and Multiple-choice Question-Answer, corresponding to VideoRefer-BenchQ.

Main Results

Interpolate start reference image.

BibTeX

@article{yuan2024videorefersuite,
  author    = {Yuqian Yuan, Hang Zhang, Wentong Li, Zesen Cheng, Boqiang Zhang, Long Li, Xin Li, Deli Zhao, Wenqiao Zhang, Yueting Zhuang, Jianke Zhu, Lidong Bing},
  title     = {VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with Video LLM},
  journal   = {arXiv},
  year      = {2024},
  url       = {http://arxiv.org/abs/2501.00599}
}