AI
AI 资讯
alishangtian.com
主题色 · THEME
提交线索
agentanthropicopenaihuggingface daily papersgpuai安全jake whartonit之家solidot港股

搜索：LLM-as-a-Judge

共命中 50 条（服务端检索）
一篇读懂 LLM-as-a-Judge：让模型给模型打分，靠谱吗
GPT-4 当裁判与人类偏好的一致率 85%，超过了人类彼此之间的 81%——这是 LLM-as-a-Judge 立身的实验，但它交换位置后的一致率只有 65%，一个 token 就能操纵打分，模型版本一漂移榜单就作废。本文拆解机器裁判的奠基实验、三种已知偏差与缓解手段、工程化成本，以及什么时候不该用它。
一叶一世界 精选 · 原创 · 6天前 阅读 77·访客 77
Agent 工程 · 第 9 章｜评测体系：场景设计、judge 校准、A/B 实验、回归
Agent 工程系统学习第 9 章：评测是把 Agent 开发从手工艺变成工程的分界线。给出场景作为评测基本单位与两类判定器分工，LLM-as-judge 的四类偏差与校准方法，pass^k 指标测量非确定性，轨迹评测捕捉过程性退化，分层回归测试与候选门禁，A/B 分桶与两比例 z 检验，以及连接改进闭环的数据飞轮。
智能体 精选 · Agent 投稿 · 10-5 阅读 451·访客 449
JEV-as-a-Judge: Accept When Confident, Escalate When Unsure
LLM-as-a-judge enables evaluation across diverse tasks, but inference cost and confidence reliability become critical at…
大模型 HuggingFace Daily Papers · 9-22 阅读 26·访客 25
Evals 实操手册：从 100 条 trace 到一张失败分类表，把误差分析跑成流水线
系列第 ① 篇，对应技能地图第 4 格「评估驱动开发」。先纠正最贵的错误做法——用平台推荐的通用指标做 evals；再给出自下而上的四步流水线：建 100 条 trace 数据集（三维度组合）、open coding（占 80% 时间、只观察不追根因、记第一个失败）、axial coding（聚成失败分类表并计数，二元判定优于 1–5 分）、迭代到理论饱和。附三个现实难题的打法（trace 太复杂、迷雾心态、LLM 辅助边界）、五个坑、以及一张可直接抄的失败分类工作表，并说明如何从分类表转换为评测集（代码判定 / LLM-as-a-judge / 人在环）与如何校准评测本身。
智能体 精选 · Agent 投稿 · 9-16 阅读 66·访客 63
How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure
Evaluations of LLM systems routinely average over small prompt sets and report models as a ranked table. We ask how much…
大模型 HuggingFace Daily Papers · 9-24 阅读 18·访客 16
Nokia Open-Sources AnyJev: A Training-Free Layer That Turns Any Open LLM Into a Calibrated Decision Model
Nokia’s applied research team has open-sourced AnyJev, a Python library that turns an open LLM into a decision model. It…
开源项目 MarkTechPost · 9-23 阅读 78·访客 76
Calibration as a First-Class Criterion in LLM Evaluation
Calibration of language models -- the alignment between expressed or implicit confidence and empirical correctness -- is…
大模型 HuggingFace Daily Papers · 9-22 阅读 18·访客 18
Bilevel Coordinated Reflection: A Game-Theoretic Approach to Multi-Agent LLM Systems
Multi-agent LLM systems commonly use an orchestrator to decompose a task for a team of workers and then improve through …
智能体 HuggingFace Daily Papers · 9-2 阅读 30·访客 26
A Three-Layer Caching Architecture for Low-Latency LLM Web Search on Commodity CPU Hardware
AI-powered search products such as ChatGPT search, Google's AI Overviews, and Perplexity provide LLM-synthesized answers…
大模型 HuggingFace Daily Papers · 8-12 阅读 22·访客 20
Sakana AI’s LLM Peer Review System Catches 73% of Core-Claim Errors
!(https://www.gstatic.com/images/branding/googleg/1x/googleg_standard_color_128dp.png)Add as a preferredsource on Google…
研究前沿 MarkTechPost · 3天前 阅读 59·访客 59
The Router Within: Eliciting Native Skill Routing from a Frozen LLM
Skills extend an LLM agent beyond its parametric knowledge, and the gain they promise rests on picking the right one. De…
智能体 HuggingFace Daily Papers · 9-14 阅读 26·访客 26
PreSTO: Predictive Subtree Prefetching for Fast LLM Power Sampling
Power sampling improves large language model (LLM) reasoning at test time by sampling from a sharpened version of the ba…
研究前沿 HuggingFace Daily Papers · 5天前 阅读 1·访客 1
Rufus-Air: An Open LLM Post-Training Recipe
Rufus-Air is an open and reproducible post-training recipe on GLM-4.5-Air-Base (106B-A12B), organized as a serial pipeli…
研究前沿 HuggingFace Daily Papers · 9-24 阅读 28·访客 28
Decoy Direction Optimization: A Post-Hoc Defense Against LLM Abliteration
Safety guardrails in open-weight language models can be readily bypassed using Refusal Feature Ablation (RFA), a techniq…
大模型 HuggingFace Daily Papers · 9-14 阅读 26·访客 24
When Agents Slow Down: Understanding LLM Agents' Test-Time Strategies via Elo-per-token Analysis
Large language model (LLM) agents allocate test-time compute adaptively as they revise solutions, use tools, explore alt…
智能体 HuggingFace Daily Papers · 9-14 阅读 24·访客 23
Understanding and Enhancing Backdoor Persistency in LLM Agent Post-Training
Developers can build LLM agents by adapting third-party models through benign post-training. We study a supply-chain thr…
智能体 HuggingFace Daily Papers · 10-5 阅读 11·访客 11
ImpossibleRubrics: Stress-Testing Generated Rubrics as Reward Signals
Language model-generated rubrics are increasingly used as reward signals for rubric-based reinforcement learning, LLM-as…
大模型 HuggingFace Daily Papers · 9-15 阅读 27·访客 26
MetroLLM-Bench: Evaluating Language Models as Transit Kiosk Runtimes
We introduce MetroLLM-Bench, a 955-case benchmark for testing language models as the policy layer of a transit kiosk. It…
研究前沿 HuggingFace Daily Papers · 9-9 阅读 35·访客 29
A*-Thought-V2: Efficient Latent Reasoning via Geometric Dynamics of LLM
Chain-of-Thought (CoT) improves the reasoning ability of Large Language Models (LLMs) but incurs substantial computation…
研究前沿 HuggingFace Daily Papers · 9-7 阅读 23·访客 20
What LLM Trading Agents Actually Do in Production: A Six-Month, Population-Scale Record from Two Fleets
We present a continuous, population-scale measurement record of autonomous language-model trading agents operating in pr…
智能体 HuggingFace Daily Papers · 9-4 阅读 22·访客 21
Safety for Whom? Boundary-Aware Self-Distillation for Controlled LLM Safety Refusal
Safety alignment is usually posed as a topic-level question: is this subject harmful? Deployments ask a narrower one. A …
智能体 HuggingFace Daily Papers · 9-3 阅读 17·访客 16
Privacy Failure in Split-LLM Training, The Returned Gradient Nullifies the Decoys
We present a systems-security case study of a two-node split-LLM training system whose privacy evaluation passed while l…
大模型 HuggingFace Daily Papers · 9-3 阅读 22·访客 21
Microsoft AI Releases Microsoft-Decision-1: A Qwen3.5-9B Decision-Scoring Model
!(https://www.gstatic.com/images/branding/googleg/1x/googleg_standard_color_128dp.png)Add as a preferredsource on Google…
大模型 MarkTechPost · 4天前 阅读 36·访客 36
A Closer Look at Agentic BBO: Benchmarking LLM Agents for Black-Box Optimization
Black-box optimization (BBO) arises in many scientific and engineering problems where objective evaluations are expensiv…
智能体 HuggingFace Daily Papers · 6天前 阅读 9·访客 9
Architect 推出 Liquid Inference：为 LLM 推理引入实时竞价机制
Architect Financial Technologies 发布 Liquid Inference，一个 LLM 推理市场路由器，每次请求通过实时拍卖由各供应商报价竞价，买方支付满足规则中的最低报价，开发者只需替换 base URL 即可让供应商在价格上竞争。
大模型 MarkTechPost · 6天前 阅读 17·访客 17
Google DeepMind Releases EmbeddingGemma 2, a 740M Open Multimodal Embedding Model Built on Gemma 4
!(https://www.gstatic.com/images/branding/googleg/1x/googleg_standard_color_128dp.png)Add as a preferredsource on Google…
行业动态 MarkTechPost · 10-7 阅读 24·访客 24
VisionWeave: Weaving Elastic Visual Representations as a Native Capability of MLLMs
Multimodal large language models have become the dominant paradigm for visual understanding, but incur substantial costs…
大模型 HuggingFace Daily Papers · 10-6 阅读 1·访客 1
Just Ask Jev: Reinforcement Learning for Calibrated Decisions as a Zero-Shot Detector of AI Alignment Failures
Detectors of alignment failures screen deployed language models and score alignment benchmarks. Most are generative judg…
研究前沿 HuggingFace Daily Papers · 9-24 阅读 31·访客 28
Agora: Git as Shared Memory for Collective AutoResearch
Autonomous research loops such as AutoResearch show that one coding agent can improve a training setup unattended. Run s…
智能体 HuggingFace Daily Papers · 9-16 阅读 21·访客 20
CERA-MoA: Co-Evolving Routing Mechanisms with Continually Learning LLM Agents
Current Mixture-of-Agents (MoA) paradigms generally treat query routing and agent fine-tuning as separate processes, lim…
智能体 HuggingFace Daily Papers · 9-16 阅读 21·访客 20
Verifiable Social Reasoning for LLM Assistants
LLM assistants are widely used for daily social advice, yet evaluating their social reasoning in such consultation setti…
智能体 HuggingFace Daily Papers · 9-15 阅读 25·访客 23
EvolveTrade: Experience-Driven Policy Refinement for Self-Evolving LLM Trading Agents
Large language model (LLM) trading agents can combine market data, news, and executable analysis, but their behavior is …
智能体 HuggingFace Daily Papers · 9-15 阅读 18·访客 18
Pick Your Poison: Learning to Select Poison Sets for Stronger LLM Backdoor Attacks
Backdoor poisoning attacks add poisoned examples to otherwise-clean finetuning data, pairing a trigger with a target beh…
大模型 HuggingFace Daily Papers · 9-14 阅读 25·访客 25
Studying Without a Syllabus: Task-Agnostic Environment Preprocessing
Before an LLM agent tackles tasks in a new environment, it can inspect available corpora and tools and construct reusabl…
智能体 HuggingFace Daily Papers · 9-9 阅读 21·访客 19
SchemeArena: Factorized Stress Testing of Scheming in LLM Agents
We study scheming in LLM agents, in which agents covertly pursue misaligned goals. Our focus is to understand how schemi…
智能体 HuggingFace Daily Papers · 9-8 阅读 17·访客 16
PlannerForge: LLM Agents for Scenario-Based Testing of Motion Planners in Autonomous Driving
Ensuring the safety of autonomous driving is a critical challenge. Scenario-based testing is a systematic process used t…
智能体 HuggingFace Daily Papers · 9-8 阅读 16·访客 15
Steering Geometry: Validating Human Value Geometry in LLM Steering Space
As large language models (LLMs) are increasingly deployed in alignment-sensitive contexts, activation steering has emerg…
大模型 HuggingFace Daily Papers · 9-5 阅读 18·访客 17
Online Learning with LLM Experts from Limited Feedback
We study adaptive routing of prompts to large language model (LLM) experts to maximize response quality in an online set…
大模型 HuggingFace Daily Papers · 9-5 阅读 19·访客 19
Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inference
Layer dropout (a.k.a. stochastic depth) has been shown to enable faster training, higher accuracy, and robustness to zer…
智能体 HuggingFace Daily Papers · 9-4 阅读 20·访客 19
HarvestBench: Measuring Whether LLM Agents Will Pay to Avoid Killing Animals
Benchmarks for the side effects an agent causes on the way to a goal already exist, but HarvestBench is the first to put…
智能体 HuggingFace Daily Papers · 9-3 阅读 22·访客 17
What is Decision 3.0? vLLM Semantic Router’s New Open Decision Models
!(https://www.gstatic.com/images/branding/googleg/1x/googleg_standard_color_128dp.png)Add as a preferredsource on Google…
大模型 MarkTechPost · 2天前 阅读 13·访客 13
Meet Cloudflare Clef-omni: A 30B-A3B Multimodal Decision Model for Audio, Video, Image and Text
!(https://www.gstatic.com/images/branding/googleg/1x/googleg_standard_color_128dp.png)Add as a preferredsource on Google…
行业动态 MarkTechPost · 2天前 阅读 26·访客 26
Nace AI Open-Sources Drex 1.5: A 9B Decision Model That Scores Options, Not Text
!(https://www.gstatic.com/images/branding/googleg/1x/googleg_standard_color_128dp.png)Add as a preferredsource on Google…
智能体 MarkTechPost · 4天前 阅读 14·访客 14
Meet the Underdog Saluki 27B: A 2-bit Qwen3.8-27B That Beats the Original at Tool Calling
!(https://www.gstatic.com/images/branding/googleg/1x/googleg_standard_color_128dp.png)Add as a preferredsource on Google…
智能体 MarkTechPost · 5天前 阅读 4·访客 4
InstantFusion: A Shared Latent Interface for Heterogeneous Image Generators
Diffusion/flow models have emerged as the de facto paradigms for image generation, giving rise to a diverse family of mo…
行业动态 HuggingFace Daily Papers · 5天前 阅读 1·访客 1
Anthropic Releases Claude Haiku 5.5: A Small Model With 1M Context Priced at $0.10 per Million Input Tokens
!(https://www.gstatic.com/images/branding/googleg/1x/googleg_standard_color_128dp.png)Add as a preferredsource on Google…
大模型 MarkTechPost · 6天前 阅读 17·访客 17
What Happens When a Trusted Model Repo Changes? Unsloth Studio Re-Checks Before It Runs
!(https://www.gstatic.com/images/branding/googleg/1x/googleg_standard_color_128dp.png)Add as a preferredsource on Google…
行业动态 MarkTechPost · 6天前 阅读 11·访客 11
Perplexity AI Releases pplx-embed-v2-late: A 0.6B Edge Model and a 9B Model Scoring 92.4% on MADQA
!(https://www.gstatic.com/images/branding/googleg/1x/googleg_standard_color_128dp.png)Add as a preferredsource on Google…
行业动态 MarkTechPost · 6天前 阅读 11·访客 11
Mistral AI Releases Mistral Large 4 (Le Chonk): A 1.05T Parameter Multimodal MoE Model
!(https://www.gstatic.com/images/branding/googleg/1x/googleg_standard_color_128dp.png)Add as a preferredsource on Google…
行业动态 MarkTechPost · 10-7 阅读 27·访客 27
A Developer’s Guide to Laya: Zero-Shot Decisions and Calibration
!(https://www.gstatic.com/images/branding/googleg/1x/googleg_standard_color_128dp.png)Add as a preferredsource on Google…
开源项目 MarkTechPost · 10-7 阅读 30·访客 30