arxiv:2609.17653
Published on Sep 15
· Submitted by
Zhengxi Lu on Sep 18
Upvote
44
Authors:
,
,
,
,
,
,
,
,
Abstract
GUI agents execute long-horizon tasks on dynamic graphical user interfaces, where pop-ups, delayed loads, and relocated widgets routinely invalidate plans fixed before execution. Recent agent-skill frameworks encapsulate reusable procedural knowledge to mitigate this, yet existing skill designs are largely developed without targeting GUI execution dynamics and treat skills as static artifacts produced before deployment rather than living procedural knowledge that improves through it. We argue that what GUI agents need is not better static skills, but skills that can be revised from execution feedback at deployment time, without additional training. We propose EvoSkill-GUI, a training-free framework in which each skill is a structured multi-file package containing retrieval metadata, executable plans, backup localization, failure-recovery rules, accessibility utilities, and failure cases. EvoSkill-GUI operates through a \emph{reflect-revise-reuse} loop: the executor performs instant in-rollout revisions, an isolated critic diagnoses failed trajectories under strict information isolation, and the executor edits specific skill files through a restricted tool interface. Across MobileWorld, AndroidWorld, and OSWorld, three mainstream GUI benchmarks spanning mobile and desktop platforms, EvoSkill-GUI consistently improves multiple base models without any training, with maximum gains of +16.2%, +6.0%, and +10.5% respectively, and evolved skill libraries continue to benefit related tasks rather than being rebuilt from scratch. Our code is available at https://github.com/ZJU-REAL/EvoSkill-GUI.
View arXiv page View PDF Project pageGitHub 17 Add to collection
Community
Paper author Paper submitter 5 days ago
We propose EvoSkill-GUI, a training-free framework in which each skill is a structured multi-file package containing retrieval metadata, executable plans, backup localization, failure-recovery rules, accessibility utilities, and failure cases.
4 days ago
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
-
From Interaction Traces to Persistent Skills: Online Evolution for Computer-Use Agents (2026)
-
SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation (2026)
-
SkillSentry: Adaptive Honey Worlds for Dynamic Safety Testing of Agent Skills (2026)
-
JailbreakSkill: Scaling Automated Red-Teaming with Reusable and Ever-Evolving Skills (2026)
-
MobilePA-Bench: Benchmarking Mobile Planner Agents on Complex Real-World Tasks (2026)
-
Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agents (2026)
-
SkillTV-Bench: Benchmarking How Well Judges Perform on Skill-Augmented Agentic Execution (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on HF Mirror checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
3 days ago
This is an automated message from the ResearchStudio team.
We created an interactive ResearchStudio Reel for this paper. It includes a visual poster, a video, and a blog, all available for download in editable formats.
Open the ResearchStudio Reel →
Download all files from HF Mirror
Please give this comment a thumbs up if you find the Reel helpful!
Want to explore or create Reels for more papers? Visit the ResearchStudio demo.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
· Sign up or log in to comment
Upvote
44
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2609.17653 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2609.17653 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2609.17653 in a Space README.md to link it from this page.