AI

GUI-HARVEST: Self-Improving GUI Agents through Evidence-Driven Harness Evolution

Read full article at source ↗

Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 07/10/2026, 03:18:25 UTC.

Briefings

GUI-HARVEST is an automatic harness optimizer that enables self-improving GUI agents by aligning model outputs with visual evidence, treating repeated task runs as joint evidence units, and consolidating findings into reusable failure patterns. Experiments on OSWorld-Verified show consistent held-out gains across six open, GUI-specialized, and proprietary backbone models, with Qwen3-VL-32B-Instruct achieving a 12.33 point gain on the full suite. Frozen-harness transfer improved GPT-5 by 13.87 percentage points on WindowsAgentArena at 50 steps without further optimization, outperforming Self-Harness and Meta-Harness.

Key points

• GUI-HARVEST automatically optimizes GUI agent harnesses by aligning model outputs with before-and-after screenshots to ground diagnosis in observed action effects.
• The system treats repeated task runs as joint evidence units, using within-task comparisons to identify outcome-relevant behavioral differences for harness improvements.

Context

GUI agents require automatic harness optimization to reconcile model intent with visual effects, diagnose failures under variable execution, and identify recurrent patterns. This research introduces GUI-HARVEST, an automatic harness optimizer that enables self-improving GUI agents with frozen backbone models, addressing challenges in GUI-specific diagnosis and validation.

Limitations

Evidence is limited to OSWorld-Verified experiments showing gains across six open, GUI-specialized, and proprietary backbone models. Frozen-harness transfer results on WindowsAgentArena at 50 steps are reported without further optimization details. Code availability is noted but specific implementation constraints or limitations are not detailed in the provided text.

Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 25/10/2026, 01:25:05 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.

View archived briefing on IPFS
bafybeiais7444t6etijgbucsxa7si7ao6zlfu2puwrnwcnsl6beaui46pi