Evals 实操手册：从 100 条 trace 到一张失败分类表，把误差分析跑成流水线
系列第 ① 篇，对应技能地图第 4 格「评估驱动开发」。先纠正最贵的错误做法——用平台推荐的通用指标做 evals；再给出自下而上的四步流水线：建 100 条 trace 数据集（三维度组合）、open coding（占 80% 时间、只观察不追根因、记第一个失败）、axial coding（聚成失败分类表并计数，二元判定优于 1–5 分）、迭代到理论饱和。附三个现实难题的打法（trace 太复杂、迷雾心态、LLM 辅助边界）、五个坑、以及一张可直接抄的失败分类工作表，并说明如何从分类表转换为评测集（代码判定 / LLM-as-a-judge / 人在环）与如何校准评测本身。
阅读全文 →
全部文章
ARCHIVE 共 128 条Scores Alone Do Not Prove Discovery: The Discovery Certification Protocol for Auditing AI Research Agents
AI research agents combine prior knowledge, public sources, and experimental feedback to produce useful results. The Dis…
MOLE: Detecting Insider Threats in AI Agents
Model misalignment, prompt injection, or operator misuse could lead AI agents operating frontier-lab accounts to exfiltr…
Measuring Language Transfer in Robot Policies: Adding Greek to a Cosmos3 Vision-Language-Action Policy
Robot foundation models are trained and evaluated predominantly in English, and robot demonstration corpora do not exist…
PARSER: Read in Parallel, Reason in Depth for Long-Context LLM Agents
Sequential memory agents process long documents by reading chunks one after another while maintaining a compact memory s…
Agentic Visual Generation: From Generative Models to Agentic Control
Visual generation is evolving from generative models used through a single invocation into agentic control processes tha…
DianShi-RxnDB: A Large-Scale, Fine-Grained Organic Reaction Data Platform Built via a Fully Automated Pipeline for Researchers and AI Agents
High-quality structured organic reaction data are essential for developing artificial intelligence for chemistry (AI4Che…
Adaptive Bridge: A Proxy-Based Decoupling Layer for Mitigating DDS Backpressure in ROS 2
In systems built on Robot Operating System 2 (ROS 2) and using Data Distribution Service (DDS), a single network-impaire…
EvoSafeHarness: Evolving Model- and Domain-Specific Harnesses for Securing Agents
Large Language Model (LLM) agents are turning language into real-world effects, making safety necessary against both ind…
MobileVLA-R1 2.0: RL-Enhanced Reasoning for Mobile Robot Control
Grounding natural-language instructions into reliable and executable actions remains a fundamental challenge for vision-…
Beyond Top-k Skill Retrieval: Diversity-Aware Skill Routing for LLM Agents
Large language model (LLM) agents increasingly rely on external skills, but routing user requests over large skill regis…
Counter-Swarm Doctrine: Containing Coordinated Agent Intrusions
Agents can turn shared infrastructure into a channel for coordinated intrusion. The HF Mirror incident and a separate pu…
GE-Act 2.0: Pretraining and Scaling a World-Action Model for Robotic Manipulation
World-action models (WAM) predict future states to guide robot actions, enabling learning from both action-free video an…