Mixture of Recursions: A model that decides when to think deeper
Introduction
Modern AI models are becoming increasingly larger and more sophisticated, but also increasingly expensive to operate. Every improvement seems to demand more memory, more GPUs, and more power. In the pursuit of trying to make better models, we might ask, Can we make AI think better without making it bigger?
The answer is fascinating. Instead of stacking more layers or adding more parameters, Mixture of Recursions(MoR) teaches a model to “think recursively”, to decide on the fly, how much effort each word deserves. Simple words get quick attention, while complex ones get deeper analysis. The result? Faster, leaner AI that performs similarly to their larger variants.
The problems with Big Models
Today’s AI models like GPT or Gemini are built on Transformers, a design that scales beautifully with size. And each boost in their intelligence comes with skyrocketing compute and memory costs. Training or even running these giants often demands specialized hardware, massive power, and corporate-scale budgets.
Recent research has tried two main tricks to make them lighter:
Parameter sharing - reusing the same layers multiple times instead of adding new ones.
Adaptive computation - letting the model decide when to stop thinking for “easy” parts.
The Mixture-of-Recursions paper steps in to fuse these two efficiency hacks into one elegant design.
The Idea Behind MoR
At its core, Mixture-of-Recursions (MoR) is about teaching an AI model to intelligently reuse its own layers. Instead of passing every word through a tall tower of layers, MoR uses a single shared block of layers again and again, like thinking in loops.
But the clever twist is that not every word needs the same amount of thinking. So MoR introduces a “router”, a lightweight decision-maker that decides how many times each word should loop through the block. Easy words might go through once. Tricky words like names, logic, or context-heavy phrases might go through several times.
This “mixture of recursions” lets the model think deeper only when needed. It saves computation, memory, and time, all while keeping performance on par with much larger models.
How It Works
The innovation lies in how MoR decides how many loops each token takes. A lightweight router analyzes token representations at each step and assigns a recursion depth per token, effectively deciding how much “thinking” it deserves.
Two routing styles mentioned are:
Expert-choice routing - each recursion step (or “expert”) selects its respective tokens to compute further.
Token-choice routing - each token chooses its full recursion path at the start.
Alongside routing, MoR refines KV caching, which determines how the model stores key-value pairs for attention. Instead of caching everything (as in standard Transformers), MoR uses:
Recursion-wise caching, which stores only the active tokens’ KV pairs at each depth, reducing I/O and memory; or
Recursive KV sharing, which reuses the first recursion’s cached keys and values across later loops, further cuts latency.
Together, these mechanisms let MoR dynamically allocate compute, reusing the same parameters while focusing attention and memory only where it matters most.
Results
Across model sizes from 135M to 1.7B parameters, MoR consistently establishes a new Pareto frontier, achieving lower validation loss and higher few-shot accuracy at the same or lower compute cost.
For example, with roughly half the parameters of a standard Transformer, MoR matched or surpassed it in benchmarks like ARC and MMLU. Under equal FLOPs, it processed more training tokens, thanks to adaptive routing that skips redundant computation.
In deployment, the benefits grow stronger. MoR’s continuous depth-wise batching, a batching method that reuses the shared block across tokens at different recursion depths, improves throughput by up to 2× (Figure 4a) compared to a vanilla model.
In essence, MoR combines the efficiency of weight-tying, the intelligence of adaptive computation, and the practicality of optimized caching, making it both faster and more resource-aware without sacrificing capability.
Conclusion
Mixture-of-Recursions represents a clear step toward smarter architectural efficiency. It shows that language models don’t always need more layers or parameters to improve; they need better control over when and where to think.
By combining recursive weight-sharing with adaptive routing and selective caching, MoR delivers high performance at a fraction of the compute cost. It’s not a radical reinvention of Transformers, but a refined evolution.
What’s ahead?
MoR changes how we think about scaling AI. Instead of endlessly building bigger models, it shows how to build smarter ones. Systems that can think deeply when needed and stay light when not.
For companies, this means cutting costs without cutting capability. For researchers, it opens a path to explore “thinking loops” that mimic how humans revisit tough problems instead of treating every thought equally.
In essence, MoR is a step toward AI that reasons more like us, adjusting its effort based on the difficulty of the task, not just the size of its brain.
