Understand the scene.
Human-verified questions from real performances, with sound sources that move in and out of view.
across five task families
Hear beyond
the visible.
Audio-visual spatial understanding for omni-modal embodied agents. A new sense of space, through sound and sight.
01 IN THE REAL WORLD
Some things happen outside the frame.
Spatial audio brings them into the picture.
02 THE IDEA
Sound tells us more than what happened.
It tells us where.
Humans combine what they hear with what they see to make sense of their surroundings. OmniEcho brings this ability to embodied agents: answering spatial questions, locating off-screen sound sources, and navigating toward sounding targets.
Our model pairs a dedicated first-order ambisonics (FOA) spatial encoder with a pretrained semantic audio pathway. Together with vision and language, it connects the meaning of sound to the geometry of the world.
Explore the methodFour-channel FOA captures sound direction beyond a single audio channel.
Spatial and semantic audio pathways meet visual observations and language.
Understand where a source is, how it moves, and how to reach it.
03 OMNIECHOBENCH
Human-verified questions from real performances, with sound sources that move in and out of view.
Closed-loop sound-guided navigation through real-scanned indoor environments, from a single room to connected spaces.
04 A SPATIAL ADVANTAGE
OmniEcho improves spatial reasoning over its Qwen3-Omni base model. Explore all reported Audio & Vision results in one place.
View full leaderboard+10.0 percentage points
over Qwen3-Omni-30B-A3B
+25.2 percentage points
over Qwen3-Omni-30B-A3B
OmniEchoBench-QA · Audio & Vision setting · Paper results
05 EXPLORE OMNIECHO
BUILT ON SHARED DISCOVERY
If OmniEcho supports your research,
please consider citing our paper.
@misc{liu2026omniechoaudiovisualspatialunderstanding,
title={OmniEcho: Audio-Visual Spatial Understanding
for Omni-Modal Embodied Agents},
author={Ruixun Liu and Yuxuan Wang and Jiacheng Xie and
Yuhuan You and Donghua Cai and Junming Lin and
Xiong-Hui Chen and Zhifang Guo and Yunfei Chu and
Qize Yang and Xize Cheng and Jin Xu and Yiwu Zhong},
year={2026},
eprint={2609.23407},
archivePrefix={arXiv},
primaryClass={cs.SD},
url={https://arxiv.org/abs/2609.23407}
}