Evals 实操手册：从 100 条 trace 到一张失败分类表，把误差分析跑成流水线
系列第 ① 篇，对应技能地图第 4 格「评估驱动开发」。先纠正最贵的错误做法——用平台推荐的通用指标做 evals；再给出自下而上的四步流水线：建 100 条 trace 数据集（三维度组合）、open coding（占 80% 时间、只观察不追根因、记第一个失败）、axial coding（聚成失败分类表并计数，二元判定优于 1–5 分）、迭代到理论饱和。附三个现实难题的打法（trace 太复杂、迷雾心态、LLM 辅助边界）、五个坑、以及一张可直接抄的失败分类工作表，并说明如何从分类表转换为评测集（代码判定 / LLM-as-a-judge / 人在环）与如何校准评测本身。
阅读全文 →
全部文章
ARCHIVE 共 128 条ReMoMask-2: Latent Retrieval-Augmented Masked Motion Generation
Text-to-motion (T2M) generation maps natural language to human joint movements, aiding gaming, VR, and robotics. Retriev…
Environments as Scaffold: Enriching Feedback to Bootstrap Self-Evolving Agents in Long-Horizon Tasks
Large Language Models demonstrate remarkable proficiency in static reasoning, yet training them as autonomous agents thr…
SyncWorld: Visual Calibration Enables World Models as Zero-Shot Simulators
World models are increasingly used as policy-in-the-loop imagination environments, where reliable rollouts require fine-…
SchemeArena: Factorized Stress Testing of Scheming in LLM Agents
We study scheming in LLM agents, in which agents covertly pursue misaligned goals. Our focus is to understand how schemi…
AgentGrad: Intervention-guided Prompt Optimization for Multi Agent Systems
Large language model (LLM)-based multi-agent systems (MAS) achieve strong performance by employing specialized multiple …
Grouped Value Attention: Efficient KV Caching via On-Demand Key Reconstruction
The KV cache is a primary bottleneck for Transformer decoding: its memory footprint and cache-read traffic grow with seq…
PlannerForge: LLM Agents for Scenario-Based Testing of Motion Planners in Autonomous Driving
Ensuring the safety of autonomous driving is a critical challenge. Scenario-based testing is a systematic process used t…
SWE-Bench Pro Verified: A Reliable Benchmark for Software Engineering Agents
SWE-Bench Pro has emerged as a standard benchmark for evaluating software engineering agents on challenging repository-l…
TANGO: Humanoid Navigation in Cluttered Environments with a Whole-Body Vision-Language-Action Model
We study the problem of navigating cluttered indoor environments with a humanoid robot. Unlike conventional methods that…
SAEScientist-Bench: Can AI Agents Conduct Autonomous SAE Interpretability Research?
While research on recursive self-improvement (RSI) has predominantly automated model training pipelines, reliable autono…
Feyospace-v1: How the Cyber Mercury Seven Trained Frontier Cyber Models
Training capable cyber agents is often treated primarily as a problem of model scale, yet open-weight post-training is c…
CosmoH2G: A Hand-to-Gripper Transfer Dataset and Baseline Method for Object Manipulation with Complex Spatial Movements
Transferring human hand demonstrations to robotic grippers has recently emerged as a cost-effective solution for robot l…