SPATIAL AUDIO · EMBODIED INTELLIGENCE

OmniEcho.

Hear beyond
the visible.

Audio-visual spatial understanding for omni-modal embodied agents. A new sense of space, through sound and sight.

Listen. Localize. Understand.
EXPLORE THE RESEARCH

Ruixun Liu1,2Yuxuan Wang3Jiacheng Xie1Yuhuan You1Donghua Cai4Junming Lin4Xiong-Hui Chen3Zhifang Guo3Yunfei Chu3Qize Yang3Xize Cheng3Jin Xu3Yiwu Zhong1,2

1,2 Peking University3 Alibaba Token Hub4 Tsinghua University

¹ School of Intelligence Science and Technology   ² State Key Laboratory of General Artificial Intelligence

197Real-world AV scenes
2,972Spatial QA pairs
900Navigation samples
30Navigation environments

01 IN THE REAL WORLD

Seeing is only
part of the story.

Some things happen outside the frame.
Spatial audio brings them into the picture.

OmniEcho / Spatial understanding demoDEMO / AUDIO + VISION
↗  Turn on sound for the full experience.Open video ↗

02 THE IDEA

A world of sound.
A sense of space.

Sound tells us more than what happened.
It tells us where.

Humans combine what they hear with what they see to make sense of their surroundings. OmniEcho brings this ability to embodied agents: answering spatial questions, locating off-screen sound sources, and navigating toward sounding targets.

Our model pairs a dedicated first-order ambisonics (FOA) spatial encoder with a pretrained semantic audio pathway. Together with vision and language, it connects the meaning of sound to the geometry of the world.

Explore the method
01 / PERCEIVE

Preserve spatial cues

Four-channel FOA captures sound direction beyond a single audio channel.

02 / CONNECT

Align sound and sight

Spatial and semantic audio pathways meet visual observations and language.

03 / ACT

Reason, then navigate

Understand where a source is, how it moves, and how to reach it.

03 OMNIECHOBENCH

Grounded in reality.
Built for spatial intelligence.

Explore the leaderboard
PERCEPTION TRACKQA / 01

Understand the scene.

Human-verified questions from real performances, with sound sources that move in and out of view.

Camera RotationSource Direction3D LocalizationSource MotionCognitive Map
2,972questions
across five task families

04 A SPATIAL ADVANTAGE

Beyond recognizing sound.
Understanding its place.

OmniEcho improves spatial reasoning over its Qwen3-Omni base model. Explore all reported Audio & Vision results in one place.

View full leaderboard
Audio & Vision · Overall28.5%

+10.0 percentage points
over Qwen3-Omni-30B-A3B

Cognitive Map47.5%

+25.2 percentage points
over Qwen3-Omni-30B-A3B

OmniEchoBench-QA · Audio & Vision setting · Paper results

05 EXPLORE OMNIECHO

Take the next step.

BUILT ON SHARED DISCOVERY

Cite our work.

If OmniEcho supports your research,
please consider citing our paper.

BIBTEX
@misc{liu2026omniechoaudiovisualspatialunderstanding,
  title={OmniEcho: Audio-Visual Spatial Understanding
         for Omni-Modal Embodied Agents},
  author={Ruixun Liu and Yuxuan Wang and Jiacheng Xie and
          Yuhuan You and Donghua Cai and Junming Lin and
          Xiong-Hui Chen and Zhifang Guo and Yunfei Chu and
          Qize Yang and Xize Cheng and Jin Xu and Yiwu Zhong},
  year={2026},
  eprint={2609.23407},
  archivePrefix={arXiv},
  primaryClass={cs.SD},
  url={https://arxiv.org/abs/2609.23407}
}