Today’s research highlights a significant trend toward optimizing human-agent and agent-agent collaboration frameworks across diverse domains. In human-computer interaction, the Computer-Use Agent (CUA) as Judge paradigm introduces a Coder-CUA collaboration framework that shifts interface design from human-centric aesthetics toward agent-native efficiency, achieving substantial improvements in task solvability and navigation success. Meanwhile, in language agent efficiency, DEPO (Dual-Efficiency Preference Optimization) addresses both step-level and trajectory-level efficiency, cutting token usage by up to 60.9% while maintaining performance. In clinical NLP, the OEMA (Ontology-Enhanced Multi-Agent Collaboration Framework) leverages SNOMED CT (Systematized Nomenclature of Medicine—Clinical Terms) to achieve near-supervised performance in zero-shot Named Entity Recognition (NER), demonstrating how multi-agent systems with structured knowledge can bridge the gap between zero-shot and supervised learning.
Total papers: 29 , Selected papers: 3
Here’s a TL;DR summary of the three papers on multi-agent collaboration and optimization:
Main Themes:
Key Insights:
Agent-Native UI Design (https://arxiv.org/abs/2511.15567v1): Proposes a Coder-CUA collaboration where coding agents design interfaces and computer-use agents evaluate them. Shows that designing UIs specifically for agents (rather than humans) improves task solvability from 67.9% to 81.5% and navigation success from 7.3% to 19.0%.
Dual-Efficiency Agent Optimization (https://arxiv.org/abs/2511.15392v1): Introduces DEPO for optimizing both step-level (tokens per step) and trajectory-level (number of steps) efficiency in LLM agents. Achieves 60.9% token reduction and 26.9% step reduction while improving performance by 29.3%.
Ontology-Guided Clinical NER (https://arxiv.org/abs/2511.15211v1): OEMA framework uses multi-agent collaboration with SNOMED CT ontology for zero-shot clinical entity recognition. Matches supervised model performance without labeled data, demonstrating how structured knowledge enhances agent reasoning.
Overall: These papers highlight a trend toward specialized agent collaboration frameworks that leverage domain knowledge and efficiency optimization, moving beyond single-agent approaches to tackle complex real-world problems.
Authors: Kevin Qinghong Lin, Siyuan Hu, Linjie Li, Zhengyuan Yang, Lijuan Wang, Philip Torr, Mike Zheng Shou
Keywords: Computer-Use Agents, Automatic GUI Design, UI Benchmark, Agent-Environment Collaboration, Task Solvability, CUA Navigation, Human-Free Evaluation
Comments: Project: https://showlab.github.io/AUI Github: https://github.com/showlab/AUI
Paper link: http://arxiv.org/abs/2511.15567v1
Computer-Use Agents (CUA) are becoming increasingly capable of autonomously operating digital environments through Graphical User Interfaces (GUI). Yet, most GUI remain designed primarily for humans–prioritizing aesthetics and usability–forcing agents to adopt human-oriented behaviors that are unnecessary for efficient task execution. At the same time, rapid advances in coding-oriented language models (Coder) have transformed automatic GUI design. This raises a fundamental question: Can CUA as judges to assist Coder for automatic GUI design? To investigate, we introduce AUI-Gym, a benchmark for Automatic GUI development spanning 52 applications across diverse domains. Using language models, we synthesize 1560 tasks that simulate real-world scenarios. To ensure task reliability, we further develop a verifier that programmatically checks whether each task is executable within its environment. Building on this, we propose a Coder-CUA in Collaboration framework: the Coder acts as Designer, generating and revising websites, while the CUA serves as Judge, evaluating functionality and refining designs. Success is measured not by visual appearance, but by task solvability and CUA navigation success rate. To turn CUA feedback into usable guidance, we design a CUA Dashboard that compresses multi-step navigation histories into concise visual summaries, offering interpretable guidance for iterative redesign. By positioning agents as both designers and judges, our framework shifts interface design toward agent-native efficiency and reliability. Our work takes a step toward shifting agents from passive use toward active participation in digital environments. Our code and dataset are available at https://github.com/showlab/AUI.
This paper introduces AUI-Gym, a benchmark for automatic GUI development, and proposes a Coder–CUA collaboration framework where coding-oriented language models (Coders) act as designers and Computer-Use Agents (CUAs) serve as judges to iteratively improve user interfaces (UIs). The key motivation is to shift UI design from human-centric aesthetics toward agent-native efficiency, enabling environments optimized for automated task execution rather than human usability.
The main contributions are threefold:
Experiments show that the framework significantly improves both function completeness (from 67.9% to 81.5% for GPT-5) and CUA success rates (e.g., from 7.3% to 19.0% for Qwen3-Coder). Task solvability feedback alone boosts functionality, while CUA navigation feedback drives agent-centric optimizations like de-stylization and clearer controls. The work demonstrates that designing UIs for agents, rather than adapting agents to human-centric interfaces, enhances both robustness and execution efficiency.
Of course. Here is a critique of the paper “Computer-Use Agents as Judges for Generative User Interface,” focusing on its strengths, weaknesses, novelty, significance, and clarity.
This paper presents a compelling vision for “agent-native” UI design by introducing a novel collaborative framework where a Coder agent designs interfaces and a Computer-Use Agent (CUA) evaluates them. The work is ambitious, well-executed, and addresses a timely and important problem. The creation of the AUI-Gym benchmark is a significant contribution in itself.
High Novelty and Conceptual Shift: The core idea is highly novel. Instead of the prevailing paradigm of adapting agents to human-centric UIs, the paper flips the problem: it adapts the UI to the agent. The “Coder-as-Designer” and “CUA-as-Judge” collaboration framework is a creative and powerful conceptual shift.
Innovative Technical Contribution (CUA Dashboard): The CUA Dashboard is an elegant solution to a critical problem: how to condense long, multi-step CUA interaction trajectories into a concise, interpretable format for the Coder. Reducing visual tokens by 76.2% while preserving essential cues is a practical and effective innovation that makes the iterative loop feasible.
Limited Exploration of the “Optimal Agent-UI”: The paper demonstrates that the framework improves UIs for a specific CUA, but it doesn’t deeply explore what an optimal agent-native UI actually looks like as a general principle. While the qualitative analysis shows trends (de-stylization, higher contrast), a more systematic set of design principles derived from the data would strengthen the contribution. Is there a risk of over-fitting the UI to the specific quirks of the CUA used for evaluation?
Human-Agent Trade-off is Acknowledged but Under-Explored: The paper briefly mentions that the resulting UIs are “optimized for agent-native efficiency and reliability,” not aesthetics. This is a critical limitation. A discussion or small experiment on how these agent-optimized UIs perform with human users would be valuable. Do they become unusable for people, or is there a beneficial middle ground? This is crucial for real-world applicability.
Cost and Scalability of the Framework: The framework relies heavily on iterative calls to powerful, expensive models (GPT-5 as Coder, Verifier, Commenter; Operator/UI-TARS as CUA). While the results are impressive, the computational cost and latency of this multi-step, iterative process are significant practical hurdles. A discussion of these limitations and potential avenues for efficiency (e.g., smaller, specialized models) would provide a more complete picture.
Ablation on the Verifier’s Role: The Verifier is a cornerstone of the benchmark’s reliability. However, its performance and potential failure modes are not deeply analyzed. How often does the GPT-5 Verifier incorrectly deem a task solvable or unsolvable? A small validation study on the Verifier’s accuracy would increase confidence in the primary results.
The results are highly significant for the fields of HCI, AI, and software automation. They convincingly show that:
This work paves the way for a future where software can self-improve its interface for automation, potentially revolutionizing areas like robotic process automation (RPA), software testing, and accessibility.
The paper is generally very well-written and clearly structured.
This is a high-quality, impactful paper that introduces a novel paradigm for UI design. Its strengths—a creative concept, a robust new benchmark, and a clever technical solution (the Dashboard)—far outweigh its weaknesses. The work makes a significant contribution by demonstrating a viable path toward environments co-designed by and for AI agents, with results that are both statistically significant and intuitively compelling. It is likely to inspire considerable follow-up research in the community.
Authors: Sirui Chen, Mengshi Zhao, Lei Xu, Yuying Zhao, Beier Zhu, Hanwang Zhang, Shengjie Zhao, Chaochao Lu
Keywords: LLM agents, preference optimization, efficiency optimization, reinforcement learning, dual-efficiency, step-level efficiency, trajectory-level efficiency
Comments: Accepted to AAAI 2026
Paper link: http://arxiv.org/abs/2511.15392v1
Recent advances in large language models (LLMs) have greatly improved their reasoning and decision-making abilities when deployed as agents. Richer reasoning, however, often comes at the cost of longer chain of thought (CoT), hampering interaction efficiency in real-world scenarios. Nevertheless, there still lacks systematic definition of LLM agent efficiency, hindering targeted improvements. To this end, we introduce dual-efficiency, comprising (i) step-level efficiency, which minimizes tokens per step, and (ii) trajectory-level efficiency, which minimizes the number of steps to complete a task. Building on this definition, we propose DEPO, a dual-efficiency preference optimization method that jointly rewards succinct responses and fewer action steps. Experiments on WebShop and BabyAI show that DEPO cuts token usage by up to 60.9% and steps by up to 26.9%, while achieving up to a 29.3% improvement in performance. DEPO also generalizes to three out-of-domain math benchmarks and retains its efficiency gains when trained on only 25% of the data. Our project page is at https://opencausalab.github.io/DEPO.
Here is a summary of the paper “DEPO: Dual-Efficiency Preference Optimization for LLM Agents”:
Key Contributions: The paper introduces the concept of “dual-efficiency” for LLM agents, comprising (1) step-level efficiency (minimizing tokens per interaction step) and (2) trajectory-level efficiency (minimizing number of steps to complete a task). This addresses limitations in current efficiency discussions that primarily focus on token reduction while ignoring the interaction dynamics of agents that require API calls or web services at each step. The authors propose DEPO, a Dual-Efficiency Preference Optimization method that explicitly targets both efficiency dimensions.
Method: DEPO extends the Kahneman-Tversky Optimization (KTO) framework by incorporating an efficiency-aware bonus into the reward function. The method uses a two-stage training process: first behavioral cloning on high-quality desirable trajectories generated via Monte Carlo Tree Search, followed by DEPO optimization that contrasts desirable and undesirable samples. The efficiency bonus rewards trajectories with fewer tokens per step and fewer total steps, applied only to desirable samples to avoid injecting unnecessary signals into low-quality data. The approach relies solely on offline preference data without requiring paired annotations, reward model training, or on-policy sampling.
Results: Experiments on WebShop and BabyAI benchmarks show that DEPO achieves significant efficiency improvements while maintaining or even enhancing performance. Compared to behavioral cloning, DEPO reduces token usage by up to 60.9% and step count by up to 26.9% over vanilla KTO, while achieving up to 29.3% improvement in success rates. The method demonstrates strong generalizability across three out-of-domain math benchmarks (GSM8K, MATH, SimulEq) and maintains efficiency gains even when trained with only 25% of the data, showing excellent sample efficiency. Ablation studies confirm that jointly optimizing both efficiency dimensions provides the best performance-efficiency trade-off.
Of course. Here is a critique of the paper “DEPO: Dual‑Efficiency Preference Optimization for LLM Agents,” focusing on its strengths and weaknesses.
This is a strong, well-executed paper that addresses a timely and practical problem: the inefficiency of Large Language Model (LLM) agents. The proposed method, DEPO, is elegant, effective, and demonstrates impressive results across multiple benchmarks. The paper is clearly written and structured.
Novel and Well-Motivated Problem Formulation: The paper’s core strength lies in its clear definition of “dual-efficiency.” By distinguishing between step-level efficiency (tokens per step) and trajectory-level efficiency (number of steps), the authors move beyond the common, narrower focus on just token count. This dual perspective is crucial for real-world agent applications where API calls and environment interactions incur significant latency and cost per step. The problem framing is both novel and highly relevant.
b(τ) that directly rewards desirable trajectories for being concise (low T_token) and direct (low T_step).α1 and α2 convincingly shows that optimizing for both step and trajectory efficiency jointly yields the best overall trade-off.Limited Comparison to State-of-the-Art Agent-Tuning Methods: While the paper compares DEPO against a strong set of base models and basic training paradigms (BC, KTO), it lacks a direct comparison to recent, more advanced agent-specific alignment methods like DMPO, StarPO, or GiGPO (which are mentioned in the related work). A comparison showing that DEPO achieves superior or comparable efficiency gains would have strengthened the claim of its contribution relative to the current state-of-the-art.
Ablation on the Data Generation Pipeline: The data generation process is complex, involving MCTS and a separate “rephrasing model” (GPT-4). While MCTS is a standard and powerful technique, the contribution of the rephrasing step is not ablated. It is unclear how much of the final performance and efficiency is attributable to DEPO versus this sophisticated, and potentially expensive, data curation process. A simpler baseline (e.g., using MCTS data without rephrasing) would have helped isolate DEPO’s specific contribution.
Hyperparameter Sensitivity: The paper shows that the α1 and α2 parameters are important, but it does not discuss how sensitive the method is to their values or provide guidance on how to set them for new tasks. The chosen values (2, 2, 3, 3) seem to work well, but a deeper analysis of the hyperparameter landscape would be valuable for practitioners.
Clarity on “Rephrasing”: The description of the rephrasing step is somewhat brief. A more detailed explanation or an example of a “Thought” before and after rephrasing would help the reader understand precisely what transformation is being applied to the data and how it contributes to step-level efficiency.
DEPO is a novel, effective, and practically valuable method for optimizing the efficiency of LLM agents. Its key contribution is the well-justified dual-efficiency framework and its simple, stable integration into an existing preference optimization algorithm. The results are significant, demonstrating dramatic reductions in token and step counts while maintaining or improving task performance. The main weaknesses are the lack of direct comparison to the most recent agent-tuning baselines and some opacity regarding the contribution of the data generation pipeline. Despite these minor points, the paper presents a compelling solution to a critical problem and is likely to influence future work on efficient LLM agents.
Authors: Xinli Tao, Xin Dong, Xuezhong Zhou
Keywords: Clinical Named Entity Recognition, Multi-Agent Systems, Zero-Shot Learning, Medical Ontology, Self-Improvement Framework, Prompt Engineering, SNOMED CT
Comments: 12 pages, 4 figures, 4 tables
Paper link: http://arxiv.org/abs/2511.15211v1
Clinical named entity recognition (NER) is crucial for extracting information from electronic health records (EHRs), but supervised models like CRF and BioClinicalBERT require costly annotated data. While zero-shot NER with large language models (LLMs) reduces this dependency, it struggles with example selection granularity and integrating prompts with self-improvement. To address this, we propose OEMA, a zero-shot clinical NER framework using multi-agent collaboration. OEMA’s three components are: a self-annotator generating examples, a discriminator filtering them via SNOMED CT, and a predictor using entity descriptions for accurate inference. On MTSamples and VAERS datasets, OEMA achieves state-of-the-art exact-match performance. Under related-match, it matches supervised BioClinicalBERT and surpasses CRF. OEMA addresses key zero-shot NER challenges through ontology-guided reasoning and multi-agent collaboration, achieving near-supervised performance and showing promise for clinical NLP applications.
Based on the provided paper, here is a summary focusing on its key contributions, methods, and results.
Key Contributions: The paper introduces OEMA (Ontology-Enhanced Multi-Agent Collaboration Framework), a novel framework designed to address two primary challenges in zero-shot Clinical Named Entity Recognition (NER): (1) the mismatch between coarse, sentence-level example selection and the fine-grained, token-level nature of the NER task, and (2) the lack of effective integration between advanced prompt engineering and self-improvement frameworks. OEMA is the first work to explicitly combine type priors with structured examples within a multi-agent system for this task.
Methods: OEMA employs a multi-agent architecture consisting of three collaborative agents:
Results: Experimental results on the MTSamples and VAERS datasets demonstrate that OEMA achieves state-of-the-art performance in zero-shot clinical NER.
Of course. Here is a critique of the paper “OEMA: Ontology-Enhanced Multi-Agent Collaboration Framework for Zero-Shot Clinical Named Entity Recognition,” covering its strengths and weaknesses.
This paper presents a well-structured and compelling study on a challenging and practically important problem: zero-shot clinical Named Entity Recognition (NER). The proposed OEMA framework is a sophisticated multi-agent system that effectively integrates ontology-guided reasoning with self-improvement mechanisms. The results are significant, demonstrating state-of-the-art performance in a zero-shot setting and approaching the performance of supervised models.
K and k provides practical insights for future implementations and shows a thoughtful design process.Computational Cost and Latency: A significant weakness, common to many LLM-based agentic systems, is the lack of discussion on computational efficiency. The three-agent pipeline, each invoking an LLM (with multiple calls for self-consistency and scoring), is undoubtedly expensive and slow. For real-time clinical applications, this cost and latency could be a major barrier to deployment. An analysis of inference time or token usage compared to simpler baselines would have been useful.
Dependence on SNOMED CT: The framework’s performance is tightly coupled with the availability and quality of the SNOMED CT ontology. The paper acknowledges this limitation, but it is a substantive one. The method’s applicability to domains without such a comprehensive, structured ontology is unclear, and its performance might degrade with less perfect ontological resources.
Scale of Evaluation: While the use of two datasets is reasonable, they are described as relatively small. Broader validation on larger, more diverse clinical corpora (e.g., MIMIC-III notes) would strengthen the claims of generalizability and robustness.
Baseline Implementation Detail: The authors note that all baselines were re-implemented using gpt-3.5-turbo-0125 to address version discrepancies. While this ensures a fair comparison, it would be helpful to briefly discuss if any hyperparameters or prompt-tuning from the original papers were retained or had to be re-optimized, to assure the reader that the baseline performances are faithfully reproduced.
Clarity on the “Discriminator” Agent: The description of the discriminator, while clear in its goal, slightly blurs the line between a “prompting strategy” and a distinct “agent.” It is essentially another LLM call with a specialized prompt for scoring. The paper could more explicitly frame the “multi-agent” aspect as a conceptual and prompt-based decomposition of the task, rather than implying separate, trained models.
This is a high-quality paper that makes a meaningful contribution to the field of zero-shot clinical NLP. Its strengths in novelty, clear problem-solving, and significant empirical results far outweigh its weaknesses. The OEMA framework presents a sophisticated and effective solution to key challenges in zero-shot NER. The primary areas for future work, as also noted by the authors, involve scaling the evaluation, reducing computational costs, and extending the framework to be less dependent on pre-existing ontologies.