Blog Post

Mixture-of-Experts (MoE): Papers to Read

August 11, 2026moellmpapersreading-list

A bottom-up reading list for understanding sparse Mixture-of-Experts models: from the 1991 origins through modern sparse LLMs (Mixtral, DeepSeek-V3, Grok), routing algorithms, load balancing, scaling laws, and the systems that make expert parallelism fast. I'll keep adding to this over time rather than starting a new post for every batch.

Type key: paper = peer-reviewed / arxiv preprint · blog = blog/post · report = technical report · book = textbook/chapter · survey = survey paper · website = tool/org site

Part 1 - Origins & Foundations

Adaptive Mixtures of Local Experts - Jacobs, Jordan, Nowlan & Hinton (1991) · the original MoE · paper
Hierarchical Mixtures of Experts and the EM Algorithm - Jordan & Jacobs (1994) · tree-structured gating trained with EM · paper
Twenty Years of Mixture of Experts - Yuksel, Wilson & Gader (2012) · survey of the classical era · survey
Learning Factored Representations in a Deep Mixture of Experts - Eigen, Ranzato & Sutskever (2013) · stacking MoE layers · paper

Part 2 - Sparse MoE for Deep Learning (2017–2021)

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer - Shazeer et al., Google (2017) · top-k gating, noisy routing, aux load-balance loss · paper
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding - Lepikhin et al., Google (2020) · 600B-param MoE translation; expert parallelism · paper
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity - Fedus, Zoph & Shazeer, Google (2021) · top-1 routing, simplified + stable · paper
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts - Du et al., Google (2021) · 1.2T params, ⅓ the training energy of GPT-3 · paper
Scaling Vision with Sparse Mixture of Experts (V-MoE) - Riquelme et al., Google (2021) · MoE for vision transformers · paper
Efficient Large Scale Language Modeling with Mixtures of Experts - Artetxe et al., Meta (2021) · paper

Part 3 - Routing Algorithms

BASE Layers: Simplifying Training of Large, Sparse Models - Lewis et al., Meta (2021) · routing as a linear assignment problem, no aux loss · paper
Hash Layers For Large Sparse Models - Roller et al., Meta (2021) · deterministic hash routing, no learned gate · paper
Mixture-of-Experts with Expert Choice Routing - Zhou et al., Google (2022) · experts pick tokens (guarantees balance) · paper
ST-MoE: Designing Stable and Transferable Sparse Expert Models - Zoph et al., Google (2022) · router z-loss, fine-tuning recipes · paper
StableMoE: Stable Routing Strategy for Mixture of Experts (2022) · addresses routing fluctuation · paper
Soft MoE: From Sparse to Soft Mixtures of Experts - Puigcerver et al., Google (2023) · fully-differentiable soft token-slot assignment · paper
Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts - Wang et al., DeepSeek (2024) · bias-based balancing without aux-loss gradient interference · paper

Part 4 - Modern Sparse LLMs

Mixtral of Experts - Jiang et al., Mistral (2024) · 8×7B, first strong open-weight MoE LLM · paper
DeepSeekMoE: Towards Ultimate Expert Specialization - Dai et al., DeepSeek (2024) · fine-grained experts + shared experts · paper
DeepSeek-V2: A Strong, Economical, and Efficient MoE Language Model - DeepSeek (2024) · 236B (21B active), MLA + DeepSeekMoE · paper
DeepSeek-V3 Technical Report - DeepSeek (2024) · 671B (37B active), aux-loss-free balancing, FP8 training · report
OLMoE: Open Mixture-of-Experts Language Models - AI2 (2024) · fully-open MoE with training data, code, logs · paper
Qwen2-MoE / Qwen1.5-MoE - Alibaba (2024) · upcycled fine-grained MoE · blog
Grok-1 Open Release - xAI (2024) · 314B MoE, open weights · blog
DBRX Technical Blog - Databricks (2024) · 132B fine-grained MoE (16 experts, 4 active) · blog
Snowflake Arctic: Efficient, Dense-MoE Hybrid - Snowflake (2024) · dense + many-expert hybrid · blog
Jamba: A Hybrid Transformer-Mamba Language Model - AI21 (2024) · MoE layers interleaved with Mamba + attention · paper
MiniMax-01: Scaling Foundation Models with Lightning Attention - MiniMax (2025) · 456B MoE + linear attention, 4M context · paper
Phi-3.5-MoE - Microsoft (2024) · 16×3.8B small-but-strong MoE · report

Part 5 - Upcycling & Merging Dense Models into MoE

Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints - Komatsuzaki et al., Google (2022) · reuse a dense model to warm-start MoE · paper
Branch-Train-Merge: Embarrassingly Parallel Training of Expert LMs - Li et al., Meta (2022) · train domain experts independently, merge · paper
Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM (BTX) - Sukhbaatar et al., Meta (2024) · combine BTM experts into a routed MoE · paper
Lory: Fully Differentiable MoE via Soft Merging of Experts (2024) · autoregressive-friendly soft expert merging · paper

Part 6 - Scaling Laws & Analysis

Unified Scaling Laws for Routed Language Models - Clark et al., DeepMind (2022) · scaling behaviour as a function of expert count · paper
Scaling Laws for Fine-Grained Mixture of Experts - Krajewski et al. (2024) · granularity as a new scaling axis · paper
Toward Understanding the Mixture-of-Experts Layer in Deep Learning - Chen et al. (2022) · theory of why routing helps · paper
A Review of Sparse Expert Models in Deep Learning - Fedus, Dean & Zoph, Google (2022) · the canonical MoE survey · survey
MoE Inference Economics / Scaling Laws (from Chinchilla to MoE) (2024) · training vs inference cost tradeoffs · paper

Part 7 - Efficiency Variants & Alternatives

Mixture-of-Depths: Dynamically Allocating Compute in Transformers - Raposo et al., DeepMind (2024) · route tokens to skip layers (depth sparsity) · paper
Mixture of a Million Experts (PEER) - He, DeepMind (2024) · product-key retrieval over huge expert pools · paper
PEER / Product Key Memory Layers - Lample et al., Meta (2019) · precursor: sparse memory access at scale · paper
LIMoE: Multimodal Contrastive Learning with a Sparse MoE - Mustafa et al., Google (2022) · one MoE for images + text · paper
From Sparse to Soft Mixtures of Experts (Soft MoE) - Google (2023) · paper
Multi-Head Mixture-of-Experts - Microsoft (2024) · split tokens into sub-tokens routed to different experts · paper

Part 8 - Systems & Serving

DeepSpeed-MoE: Advancing MoE Inference and Training to Power Next-Gen AI Scale - Rajbhandari et al., Microsoft (2022) · MoE inference optimization + distillation · paper
Tutel: Adaptive Mixture-of-Experts at Scale - Hwang et al., Microsoft (2022) · dynamic parallelism switching · paper
MegaBlocks: Efficient Sparse Training with Mixture-of-Experts - Gale et al., Stanford (2022) · block-sparse GPU kernels, no token dropping · paper
FasterMoE: Modeling and Optimizing Training of Large-Scale Dynamic MoE - He et al., Tsinghua (2022) · paper
Comet: Fine-grained Computation-Communication Overlap for MoE - ByteDance (2025) · overlapping expert-parallel all-to-all with compute · paper
DeepSeek-V3 DualPipe & Expert Parallel Load Balancer - DeepSeek (2024) · production-scale MoE systems engineering · report
Megatron-LM / Megatron-Core MoE - NVIDIA (2019, MoE support later) · tensor + expert parallelism · paper

Part 9 - Surveys & Big-Picture Reading

A Review of Sparse Expert Models in Deep Learning - Fedus, Dean & Zoph (2022) · survey
A Survey on Mixture of Experts in Large Language Models - Cai et al. (2024) · algorithm + system + application taxonomy · survey
Mixture of Experts Explained - Hugging Face (2023) · accessible intro with diagrams · blog
Switch Transformers & Sparsity (Google AI Blog) - Google (2021) · blog
How DeepSeek-V3 Trains a 671B MoE for $5.5M - DeepSeek (2024) · read alongside the systems section · report
The Bitter Lesson - Rich Sutton (2019) · conditional-compute framing for why sparse scaling wins · blog