arxiv:2609.19499
Published on Sep 16
· Submitted by
Mobina Kashaniyan on Sep 18
Upvote
36
Authors:
Abstract
Test-time scaling can improve large language model reasoning by generating and combining multiple candidate responses. In sampling-based methods, the inference budget is often described by the number of generated candidates, N. However, N tells us how many candidates are generated, not how they are executed. The same candidate budget can be produced in one batched generation call or split across several sequential calls with smaller batch sizes. We first study the effect of increasing N on reasoning accuracy using Phi-3-mini and Qwen2.5-1.5B on 500 GSM8K prompts. As expected, increasing N from 1 to 8 improves accuracy by 8.4 percentage points for Phi-3-mini and 18.4 points for Qwen2.5-1.5B. However, accuracy alone does not show the systems cost of using a larger candidate budget. We therefore fix N = 8 and compare four generation schedules: 1x8, 2x4, 4x2, and 8x1, where axb denotes a generation calls with b candidates per call. We measure latency, throughput, GPU-hours, and gross GPU-device energy while keeping the total candidate count fixed. On A100 GPUs, eight serial calls use 4.64-4.86x as much gross GPU-device energy and have 5.77-6.12x the P95 latency of one batched call with eight candidates. The same pattern appears across three independently scheduled A100 nodes per model and in short-output SciQ/V100 experiments. These results show that candidate count alone is not enough to describe the systems cost of multi-candidate test-time scaling. When candidates are independent and memory allows it, fewer generation calls with larger batch sizes are more efficient. Evaluations should therefore report not only candidate count and accuracy, but also generation schedule and GPU-level systems metrics.
View arXiv page View PDF Add to collection
Community
Paper author Paper submitter 4 days ago
At fixed candidate budget N=8, how you generate candidates matters as much as how many you generate. On A100 GPUs, serial 8×1 uses ~4.6–4.9× more GPU energy and ~5.8–6.1× higher P95 latency than batched 1×8, while keeping the same candidate count. Takeaway: report generation schedule (calls × batch), not just N. Accepted to EESP@SC26. Paper: https://arxiv.org/abs/2609.19499
4 days ago
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
-
Characterization of Request and Token Energy Costs for LLM Inference Workloads on GPU Platforms (2026)
-
Interpretable Adaptive Sampling for LLM Test-Time Scaling (2026)
-
A Measurement Study of LLM Inference Trade-offs Across Edge Continuum Hardware (2026)
-
CoBa: Cost-Effective Test-Time Scaling via Compute-Balanced Routing (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on HF Mirror checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
3 days ago
Everyone quotes test-time scaling curves like N is the only knob. It isn't — the same N costs me wildly different GPU-hours depending on whether I batch candidates or drain them sequentially, and the accuracy delta isn't trivial either. That's the number I actually need when I size a cluster, and it's almost never in the abstract. I'd love to see this extended past a single model family, because the batch-vs-sequential gap is probably where the real cost surprises hide. Until then, I'm treating any scaling claim without a generation-strategy footnote as incomplete.
3 days ago
Keep it up, Iowa State!
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
· Sign up or log in to comment
Upvote
36
Get this paper in your agent:
hf papers read 2609.19499
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2609.19499 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2609.19499 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2609.19499 in a Space README.md to link it from this page.