According to the official WeChat account of “Tongyi Qwen Qwen”, the main features of this model are as follows:
All-round innovative architecture: The Qwen team proposed a new Thinker-Talker architecture, an end-to-end multi-modal model designed to support cross-modal understanding of text/image/audio/video while generating text and natural speech responses in a streaming manner. Qwen proposed a new position encoding technology called TMRoPE (Time-alignedMultimodalRoPE), which achieves precise synchronization of video and audio input through timeline alignment.
Real-time audio and video interaction: The architecture is designed to support fully real-time interaction, supporting chunked input and instant output.
Natural-smooth speech generation: Surpasses many existing streaming and non-streaming alternatives in naturalness and stability of speech generation.
Full-modal performance advantage: Excellent performance when benchmarking single-modal models of the same size. The Qwen2.5-Omni outperforms the similarly sized Qwen2-Audio in audio capabilities and is on par with the Qwen2.5-VL-7B.
Excellent end-to-end voice command following ability: Qwen2.5-Omni shows an effect comparable to text input processing in end-to-end voice command following, and performs well in benchmark tests such as MMLU general knowledge understanding and GSM8K mathematical reasoning.
Qwen2.5-Omni adopts Thinker-Talker dual-core architecture. The Thinker module is like a brain, responsible for processing multi-modal inputs such as text, audio, and video, and generating high-level semantic representations and corresponding text content; the Talker module is similar to a vocal organ, receiving the semantic representation and text output by Thinker in real-time in a streaming manner, and smoothly synthesizing discrete speech units. Thinker is based on the Transformer decoder architecture and integrates audio/image encoders for feature extraction; Talker uses a dual-track autoregressive Transformer decoder design to directly receive high-dimensional representations from Thinker during training and inference, and share all historical context information to form an end-to-end unified model architecture.
Model architecture diagram
In terms of model performance, Qwen2.5-Omni performs better than similar-sized single-modal models and closed-source models, such as Qwen2.5-VL-7B, Qwen2-Audio and Gemini-1.5-pro, in various modes including images, audio, audio and video.
In the multi-modal task OmniBench, Qwen2.5-Omni achieved SOTA performance. In addition, in single-modal tasks, Qwen2.5-Omni performs well in multiple fields, including speech recognition (CommonVoice), translation (CoVoST2), audio understanding (MMAU), image reasoning (MMMU, MMStar), video understanding (MVBench), and speech generation (Seed-tts-eval and subjective natural hearing).
The model is now open source on HuggingFace, ModelScope, DashScope and GitHub.