OmniGUI: Benchmarking GUI Agents in Omni-Modal Smartphone Environments
The paper introduces OmniGUI, a new benchmark for evaluating graphical user interface (GUI) agents in omni-modal smartphone environments. Unlike traditional benchmarks that rely on static screenshots, OmniGUI incorporates multimodal inputs such as audio and video. The study reveals that current models struggle with tasks requiring synchronous temporal and auditory signals, highlighting the need for improved omni-modal frameworks.
- ▪OmniGUI is the first step-level benchmark designed for GUI agents in omni-modal smartphone environments.
- ▪The dataset includes 709 expert-demonstrated episodes with 2,579 action steps across 29 applications.
- ▪Current models perform well on visually static tasks but show degraded performance in environments with temporal and auditory signals.
Opening excerpt (first ~120 words) tap to expand
Computer Science > Human-Computer Interaction arXiv:2605.18758 (cs) [Submitted on 3 Apr 2026] Title:OmniGUI: Benchmarking GUI Agents in Omni-Modal Smartphone Environments Authors:Felix Henry, Xiaochen Lin, Jiangyou Zhu, Yangfan, Bingqian Zhang, Min Chen, Shiyu Huang View a PDF of the paper titled OmniGUI: Benchmarking GUI Agents in Omni-Modal Smartphone Environments, by Felix Henry and 6 other authors View PDF HTML (experimental) Abstract:Current benchmarks for graphical user interface (GUI) agents predominantly rely on static screenshots. However, real-world smartphone interaction routinely requires agents to process transient audio cues and temporal video dynamics that are tightly coupled with the moment of action.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at arXiv cs.AI.
Discussion
0 commentsMore from arXiv cs.AI
