The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction

Papers

arxiv:2609.18063

Published on Sep 16

· Submitted by

yumu on Sep 17

· Edge0

Upvote

18

Authors:

Yu Lin ,

Yiming Wang ,

Runyuan Cai ,

,

Abstract

Mixture-of-experts (MoE) inference on consumer hardware is bounded by weight memory: a 35B-class model is 19.5GB at 4-bit, and sparsity shrinks the compute per token, not the bytes that must be held. Naive offloading to SSD does not help on its own, because layer N+1's experts must be chosen before layer N's output exists, so the reads cannot start early enough to hide behind compute. We present Edge0, a streaming MoE inference engine that closes the gap with a prerouter: a per-layer head predicts the next layer's routing one token ahead, and the prediction is consumed as the routing itself, so the staged expert set equals the routed set and nothing is dropped. An unmerged recovery LoRA, trained on the student path, pays back the quality lost to int4 quantization and routing replacement. On a single 24GB machine, Edge0 serves a 35B MoE at 20tok/s inside 3GiB of peak active memory, within a few points of its fp16 teacher on average across five public benchmarks. An 8B tier runs on the same framework, and the framework, checkpoints, and adapters are open source.

View arXiv page View PDF GitHub 2.05k Add to collection

Community

bupalinyu

Paper author Paper submitter 6 days ago

Mixture-of-experts (MoE) inference on consumer hardware is bounded by weight memory: a 35B-class model is 19.5GB at 4-bit, and sparsity shrinks the compute per token, not the bytes that must be held. Naive offloading to SSD does not help on its own, because layer N+1's experts must be chosen before layer N's output exists, so the reads cannot start early enough to hide behind compute. We present Edge0, a streaming MoE inference engine that closes the gap with a prerouter: a per-layer head predicts the next layer's routing one token ahead, and the prediction is consumed as the routing itself, so the staged expert set equals the routed set and nothing is dropped. An unmerged recovery LoRA, trained on the student path, pays back the quality lost to int4 quantization and routing replacement. On a single 24GB machine, Edge0 serves a 35B MoE at 20tok/s inside 3GiB of peak active memory, within a few points of its fp16 teacher on average across five public benchmarks. An 8B tier runs on the same framework, and the framework, checkpoints, and adapters are open source.

O96a

5 days ago

Does a wrong prediction stall the decode to fetch the right experts, or push forward with the wrong ones and eat the quality hit? That's the whole game with one-token-ahead routing prediction. If it's the latter, I want the accuracy curve, not the average — the tail. One bad prediction mid-generation silently degrades output while your latency numbers still look fine. And on a consumer box, a wrong SSD fetch can cost more than the prediction saved. What's the per-layer accuracy, and what does the latency distribution actually look like?

·

bupalinyu

Paper author 5 days ago

•

edited 5 days ago

we use prediction result as routing, so it is 100% accuracy.

librarian-bot

5 days ago

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

• Cacheable by Design? Training Mixture-of-Experts Routers for Locality Against the Edge Memory-Bandwidth Wall: A Pre-Registered Negative Result with a Systems Measurement Study (2026)

• Budgeting Bytes: A Windowed Storage Roofline and Dual-Budget Architecture Ablations for Storage-Bound LLM Decoding (2026)

• FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution (2026)

• NeuroPrefetcher: Storage-Aware Sparse LLM Inference via Delta Prefetching (2026)

• Who Should Own the Expert Cache? Kernel-Managed Tiering for Trillion-Parameter MoE Inference (2026)

• HiSparse: Scaling Sparse-Attention Decoding with Hierarchical KV Cache Management (2026)

• Q-First: Most of Attention Needs Only the Query in Disaggregated LLM Decoding (2026)

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on HF Mirror checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment

Upvote

18

Get this paper in your agent:

hf papers read 2609.18063

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 3

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2609.18063 in a dataset README.md to link it from this page.

Spaces citing this paper 2

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.