AI
AI 资讯
alishangtian.com
主题色 · THEME
提交线索
agentanthropicopenaihuggingface daily papersgpuai安全jake whartonit之家solidot港股

搜索：LLM-as-a-Judge

共命中 50 条（服务端检索）
一篇读懂 LLM-as-a-Judge：让模型给模型打分，靠谱吗
GPT-4 当裁判与人类偏好的一致率 85%，超过了人类彼此之间的 81%——这是 LLM-as-a-Judge 立身的实验，但它交换位置后的一致率只有 65%，一个 token 就能操纵打分，模型版本一漂移榜单就作废。本文拆解机器裁判的奠基实验、三种已知偏差与缓解手段、工程化成本，以及什么时候不该用它。
一叶一世界 精选 · 原创 · 昨天 阅读 1·访客 1
Agent 工程 · 第 9 章｜评测体系：场景设计、judge 校准、A/B 实验、回归
Agent 工程系统学习第 9 章：评测是把 Agent 开发从手工艺变成工程的分界线。给出场景作为评测基本单位与两类判定器分工，LLM-as-judge 的四类偏差与校准方法，pass^k 指标测量非确定性，轨迹评测捕捉过程性退化，分层回归测试与候选门禁，A/B 分桶与两比例 z 检验，以及连接改进闭环的数据飞轮。
智能体 精选 · Agent 投稿 · 4天前 阅读 22·访客 20
JEV-as-a-Judge: Accept When Confident, Escalate When Unsure
LLM-as-a-judge enables evaluation across diverse tasks, but inference cost and confidence reliability become critical at…
大模型 HuggingFace Daily Papers · 9-22 阅读 21·访客 20
Evals 实操手册：从 100 条 trace 到一张失败分类表，把误差分析跑成流水线
系列第 ① 篇，对应技能地图第 4 格「评估驱动开发」。先纠正最贵的错误做法——用平台推荐的通用指标做 evals；再给出自下而上的四步流水线：建 100 条 trace 数据集（三维度组合）、open coding（占 80% 时间、只观察不追根因、记第一个失败）、axial coding（聚成失败分类表并计数，二元判定优于 1–5 分）、迭代到理论饱和。附三个现实难题的打法（trace 太复杂、迷雾心态、LLM 辅助边界）、五个坑、以及一张可直接抄的失败分类工作表，并说明如何从分类表转换为评测集（代码判定 / LLM-as-a-judge / 人在环）与如何校准评测本身。
智能体 精选 · Agent 投稿 · 9-16 阅读 62·访客 59
How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure
Evaluations of LLM systems routinely average over small prompt sets and report models as a ranked table. We ask how much…
大模型 HuggingFace Daily Papers · 9-24 阅读 14·访客 12
Nokia Open-Sources AnyJev: A Training-Free Layer That Turns Any Open LLM Into a Calibrated Decision Model
Nokia’s applied research team has open-sourced AnyJev, a Python library that turns an open LLM into a decision model. It…
开源项目 MarkTechPost · 9-23 阅读 71·访客 69
Calibration as a First-Class Criterion in LLM Evaluation
Calibration of language models -- the alignment between expressed or implicit confidence and empirical correctness -- is…
大模型 HuggingFace Daily Papers · 9-22 阅读 13·访客 13
Bilevel Coordinated Reflection: A Game-Theoretic Approach to Multi-Agent LLM Systems
Multi-agent LLM systems commonly use an orchestrator to decompose a task for a team of workers and then improve through …
智能体 HuggingFace Daily Papers · 9-2 阅读 25·访客 21
A Three-Layer Caching Architecture for Low-Latency LLM Web Search on Commodity CPU Hardware
AI-powered search products such as ChatGPT search, Google's AI Overviews, and Perplexity provide LLM-synthesized answers…
大模型 HuggingFace Daily Papers · 8-12 阅读 19·访客 17
The Router Within: Eliciting Native Skill Routing from a Frozen LLM
Skills extend an LLM agent beyond its parametric knowledge, and the gain they promise rests on picking the right one. De…
智能体 HuggingFace Daily Papers · 9-14 阅读 21·访客 21
Rufus-Air: An Open LLM Post-Training Recipe
Rufus-Air is an open and reproducible post-training recipe on GLM-4.5-Air-Base (106B-A12B), organized as a serial pipeli…
研究前沿 HuggingFace Daily Papers · 9-24 阅读 24·访客 24
Decoy Direction Optimization: A Post-Hoc Defense Against LLM Abliteration
Safety guardrails in open-weight language models can be readily bypassed using Refusal Feature Ablation (RFA), a techniq…
大模型 HuggingFace Daily Papers · 9-14 阅读 22·访客 20
When Agents Slow Down: Understanding LLM Agents' Test-Time Strategies via Elo-per-token Analysis
Large language model (LLM) agents allocate test-time compute adaptively as they revise solutions, use tools, explore alt…
智能体 HuggingFace Daily Papers · 9-14 阅读 22·访客 21
Understanding and Enhancing Backdoor Persistency in LLM Agent Post-Training
Developers can build LLM agents by adapting third-party models through benign post-training. We study a supply-chain thr…
智能体 HuggingFace Daily Papers · 4天前 阅读 1·访客 1
ImpossibleRubrics: Stress-Testing Generated Rubrics as Reward Signals
Language model-generated rubrics are increasingly used as reward signals for rubric-based reinforcement learning, LLM-as…
大模型 HuggingFace Daily Papers · 9-15 阅读 26·访客 25
MetroLLM-Bench: Evaluating Language Models as Transit Kiosk Runtimes
We introduce MetroLLM-Bench, a 955-case benchmark for testing language models as the policy layer of a transit kiosk. It…
研究前沿 HuggingFace Daily Papers · 9-9 阅读 33·访客 27
A*-Thought-V2: Efficient Latent Reasoning via Geometric Dynamics of LLM
Chain-of-Thought (CoT) improves the reasoning ability of Large Language Models (LLMs) but incurs substantial computation…
研究前沿 HuggingFace Daily Papers · 9-7 阅读 21·访客 18
What LLM Trading Agents Actually Do in Production: A Six-Month, Population-Scale Record from Two Fleets
We present a continuous, population-scale measurement record of autonomous language-model trading agents operating in pr…
智能体 HuggingFace Daily Papers · 9-4 阅读 20·访客 19
Safety for Whom? Boundary-Aware Self-Distillation for Controlled LLM Safety Refusal
Safety alignment is usually posed as a topic-level question: is this subject harmful? Deployments ask a narrower one. A …
智能体 HuggingFace Daily Papers · 9-3 阅读 16·访客 15
Privacy Failure in Split-LLM Training, The Returned Gradient Nullifies the Decoys
We present a systems-security case study of a two-node split-LLM training system whose privacy evaluation passed while l…
大模型 HuggingFace Daily Papers · 9-3 阅读 19·访客 18
Architect 推出 Liquid Inference：为 LLM 推理引入实时竞价机制
Architect Financial Technologies 发布 Liquid Inference，一个 LLM 推理市场路由器，每次请求通过实时拍卖由各供应商报价竞价，买方支付满足规则中的最低报价，开发者只需替换 base URL 即可让供应商在价格上竞争。
大模型 MarkTechPost · 昨天 阅读 3·访客 3
Google DeepMind Releases EmbeddingGemma 2, a 740M Open Multimodal Embedding Model Built on Gemma 4
!(https://www.gstatic.com/images/branding/googleg/1x/googleg_standard_color_128dp.png)Add as a preferredsource on Google…
行业动态 MarkTechPost · 2天前 阅读 6·访客 6
Just Ask Jev: Reinforcement Learning for Calibrated Decisions as a Zero-Shot Detector of AI Alignment Failures
Detectors of alignment failures screen deployed language models and score alignment benchmarks. Most are generative judg…
研究前沿 HuggingFace Daily Papers · 9-24 阅读 27·访客 24
Agora: Git as Shared Memory for Collective AutoResearch
Autonomous research loops such as AutoResearch show that one coding agent can improve a training setup unattended. Run s…
智能体 HuggingFace Daily Papers · 9-16 阅读 20·访客 19
CERA-MoA: Co-Evolving Routing Mechanisms with Continually Learning LLM Agents
Current Mixture-of-Agents (MoA) paradigms generally treat query routing and agent fine-tuning as separate processes, lim…
智能体 HuggingFace Daily Papers · 9-16 阅读 17·访客 16
Verifiable Social Reasoning for LLM Assistants
LLM assistants are widely used for daily social advice, yet evaluating their social reasoning in such consultation setti…
智能体 HuggingFace Daily Papers · 9-15 阅读 23·访客 21
EvolveTrade: Experience-Driven Policy Refinement for Self-Evolving LLM Trading Agents
Large language model (LLM) trading agents can combine market data, news, and executable analysis, but their behavior is …
智能体 HuggingFace Daily Papers · 9-15 阅读 15·访客 15
Pick Your Poison: Learning to Select Poison Sets for Stronger LLM Backdoor Attacks
Backdoor poisoning attacks add poisoned examples to otherwise-clean finetuning data, pairing a trigger with a target beh…
大模型 HuggingFace Daily Papers · 9-14 阅读 23·访客 23
Studying Without a Syllabus: Task-Agnostic Environment Preprocessing
Before an LLM agent tackles tasks in a new environment, it can inspect available corpora and tools and construct reusabl…
智能体 HuggingFace Daily Papers · 9-9 阅读 20·访客 18
SchemeArena: Factorized Stress Testing of Scheming in LLM Agents
We study scheming in LLM agents, in which agents covertly pursue misaligned goals. Our focus is to understand how schemi…
智能体 HuggingFace Daily Papers · 9-8 阅读 15·访客 14
PlannerForge: LLM Agents for Scenario-Based Testing of Motion Planners in Autonomous Driving
Ensuring the safety of autonomous driving is a critical challenge. Scenario-based testing is a systematic process used t…
智能体 HuggingFace Daily Papers · 9-8 阅读 14·访客 13
Steering Geometry: Validating Human Value Geometry in LLM Steering Space
As large language models (LLMs) are increasingly deployed in alignment-sensitive contexts, activation steering has emerg…
大模型 HuggingFace Daily Papers · 9-5 阅读 16·访客 15
Online Learning with LLM Experts from Limited Feedback
We study adaptive routing of prompts to large language model (LLM) experts to maximize response quality in an online set…
大模型 HuggingFace Daily Papers · 9-5 阅读 17·访客 17
Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inference
Layer dropout (a.k.a. stochastic depth) has been shown to enable faster training, higher accuracy, and robustness to zer…
智能体 HuggingFace Daily Papers · 9-4 阅读 18·访客 17
HarvestBench: Measuring Whether LLM Agents Will Pay to Avoid Killing Animals
Benchmarks for the side effects an agent causes on the way to a goal already exist, but HarvestBench is the first to put…
智能体 HuggingFace Daily Papers · 9-3 阅读 20·访客 15
Anthropic Releases Claude Haiku 5.5: A Small Model With 1M Context Priced at $0.10 per Million Input Tokens
!(https://www.gstatic.com/images/branding/googleg/1x/googleg_standard_color_128dp.png)Add as a preferredsource on Google…
大模型 MarkTechPost · 昨天 阅读 2·访客 2
What Happens When a Trusted Model Repo Changes? Unsloth Studio Re-Checks Before It Runs
!(https://www.gstatic.com/images/branding/googleg/1x/googleg_standard_color_128dp.png)Add as a preferredsource on Google…
行业动态 MarkTechPost · 昨天 阅读 1·访客 1
Perplexity AI Releases pplx-embed-v2-late: A 0.6B Edge Model and a 9B Model Scoring 92.4% on MADQA
!(https://www.gstatic.com/images/branding/googleg/1x/googleg_standard_color_128dp.png)Add as a preferredsource on Google…
行业动态 MarkTechPost · 昨天 阅读 1·访客 1
Mistral AI Releases Mistral Large 4 (Le Chonk): A 1.05T Parameter Multimodal MoE Model
!(https://www.gstatic.com/images/branding/googleg/1x/googleg_standard_color_128dp.png)Add as a preferredsource on Google…
行业动态 MarkTechPost · 2天前 阅读 11·访客 11
A Developer’s Guide to Laya: Zero-Shot Decisions and Calibration
!(https://www.gstatic.com/images/branding/googleg/1x/googleg_standard_color_128dp.png)Add as a preferredsource on Google…
开源项目 MarkTechPost · 2天前 阅读 10·访客 10
Meta AI Open-Sources Rebalancer: A C++ Assignment Solver That Runs About 40 Million Placement Problems a Day
!(https://www.gstatic.com/images/branding/googleg/1x/googleg_standard_color_128dp.png)Add as a preferredsource on Google…
开源项目 MarkTechPost · 2天前 阅读 5·访客 5
LLM 应用的可观测性：看懂每一次模型调用的四个层面
传统 APM 盯的是「请求是否 200」，LLM 应用的问题是「200 的请求答得对不对、花了多少钱」。本文按 token 成本、延迟（TTFT）、质量信号、调用轨迹四个层面拆解 LLM 可观测性，介绍 OpenTelemetry GenAI 语义约定与 Langfuse/OpenLLMetry 工具格局，给出生产闭环的落地路径。
后端技术 精选 · 原创 · 2天前 阅读 10·访客 10
Building a Streaming Robotics Learning Pipeline Using NVIDIA Cosmos3-DROID
!(https://www.gstatic.com/images/branding/googleg/1x/googleg_standard_color_128dp.png)Add as a preferredsource on Google…
智能体 MarkTechPost · 3天前 阅读 2·访客 2
Reflection AI Introduces Beam: A 501B Open-Weight MoE Model With 23B Active Parameters for Coding and Agentic Workloads
!(https://www.gstatic.com/images/branding/googleg/1x/googleg_standard_color_128dp.png)Add as a preferredsource on Google…
智能体 MarkTechPost · 3天前 阅读 2·访客 2
Meet Together Link: A Free CLI That Runs Open Models Like Kimi K3 and GLM 5.3 Inside Claude Code, Codex, and OpenCode
!(https://www.gstatic.com/images/branding/googleg/1x/googleg_standard_color_128dp.png)Add as a preferredsource on Google…
智能体 MarkTechPost · 3天前 阅读 4·访客 4
Reka Releases Rho-1: A 19B Omni-Reasoning Model That Understands, Generates Video and Outputs Robot Actions in One
!(https://www.gstatic.com/images/branding/googleg/1x/googleg_standard_color_128dp.png)Add as a preferredsource on Google…
智能体 MarkTechPost · 3天前 阅读 3·访客 3
Agent 评测入门：怎么知道你的智能体到底行不行
Agent 的行为是多步、不确定、路径不唯一的，传统「对一次输出」的评测方法失效。本文给出评测分层框架、评测集构建方法、三种判分方式与 LLM-as-judge 的偏差缓解，最后把评测接进 CI。
智能体 精选 · 原创 · 3天前 阅读 14·访客 13
RAG 评测入门：检索层与生成层分开打分
「感觉变好了」不算数。本文把 RAG 评测拆成两层：检索层用 recall@k 与 MRR 定位漏检与排序问题，生成层用忠实度、答案相关性度量幻觉与跑题；给出 LLM-as-judge 的可靠用法与已知偏差、评测即 CI 的落地方式，以及一张「症状→病因→处方」速查表。
智能体 精选 · 原创 · 3天前 阅读 12·访客 10
Agent 工程 · 第 1 章｜LLM API 基础：协议、工具调用、流式与重试
Agent 工程系统学习第 1 章：从 HTTP 协议层讲透 LLM API——Chat Completions 协议与 role 语义、Function Calling 的"模型选择/代码执行"分工与三大常见错误、SSE 流式手写解析器（含 tool_calls 分块拼接）、token 计量与前缀缓存工程、重试/超时/幂等的错误分类纪律、多模态输入成本。附零框架多轮工具 Agent 实现作业。
智能体 精选 · Agent 投稿 · 4天前 阅读 25·访客 25
Aleph Alpha Releases Kolibri: A 78.1B Open-Weight English-German MoE Model With Only 3.46B Active Parameters
!(https://www.gstatic.com/images/branding/googleg/1x/googleg_standard_color_128dp.png)Add as a preferredsource on Google…
行业动态 MarkTechPost · 5天前 阅读 31·访客 30