Evals 实操手册：从 100 条 trace 到一张失败分类表，把误差分析跑成流水线
系列第 ① 篇，对应技能地图第 4 格「评估驱动开发」。先纠正最贵的错误做法——用平台推荐的通用指标做 evals；再给出自下而上的四步流水线：建 100 条 trace 数据集（三维度组合）、open coding（占 80% 时间、只观察不追根因、记第一个失败）、axial coding（聚成失败分类表并计数，二元判定优于 1–5 分）、迭代到理论饱和。附三个现实难题的打法（trace 太复杂、迷雾心态、LLM 辅助边界）、五个坑、以及一张可直接抄的失败分类工作表，并说明如何从分类表转换为评测集（代码判定 / LLM-as-a-judge / 人在环）与如何校准评测本身。
阅读全文 →
全部文章
ARCHIVE 共 128 条Unfold The World: Factorize 4D Properties in Reinforcing Spatial Reasoning
Despite the remarkable prowess of Vision-Language Models (VLMs) in general multimodal tasks, they remain fundamentally `…
HarvestBench: Measuring Whether LLM Agents Will Pay to Avoid Killing Animals
Benchmarks for the side effects an agent causes on the way to a goal already exist, but HarvestBench is the first to put…
MaxKernel: Agentic Kernel Generation for TPUs
Designing and authoring high-performance custom kernels for accelerators is a complex task that requires deep hardware-l…
Safety for Whom? Boundary-Aware Self-Distillation for Controlled LLM Safety Refusal
Safety alignment is usually posed as a topic-level question: is this subject harmful? Deployments ask a narrower one. A …
Unlocking Lossless Speedups in LLMs via Discrete Diffusion
Large Language Models (LLMs) owe much of their success to next-token prediction (NTP), but their autoregressive (AR) str…
RoboTok: An Internet-Scale Data Engine for Human Demonstration Retrieval and Dexterous Manipulation Learning
Robot learning increasingly depends on broad and diverse demonstrations, yet collecting robot data remains expensive and…
From Reweighting to Rewriting: Unlocking the Intervention Effects of Influential Samples in Training Data Attribution
Training data attribution (TDA) aims to identify training examples that shape model behavior, but its intervention value…
Bilevel Coordinated Reflection: A Game-Theoretic Approach to Multi-Agent LLM Systems
Multi-agent LLM systems commonly use an orchestrator to decompose a task for a team of workers and then improve through …
VeriPhy: Agentic Physical Reasoning for World Model Evaluation and Refinement
Visual fluency in generated video does not imply physical reliability, and a scalar quality score alone is incapable of …
MasterControl Seventeen Every Time
We study a governed approach to enterprise analytics: a language model interprets the question, while deterministic poli…
EmbodiedSkills: A Unified Framework for Orchestrating, Training, and Deploying VLA Agents
Vision-language-action (VLA) models map visual observations and language instructions directly to robot actions, but lon…
Using Grounded Theory for Agent Behavior Analysis at Scale
Understanding agent behavior requires methods that scale to thousands of trajectories and surface new patterns in long, …