I was going through an insightful blog about how Gradient Labs built their customer-facing agentic system. One part that particularly caught my attention was their failover mechanism.

Here, when a single request takes more than a threshold time, it triggers the failover mechanism, which allows them to switch their LLM provider or model depending on the severity of the request timeout.

However, in the example they shared, between 8:00 and 9:00, the mean latency and p75 latency (75th percentile) showed an increase. The entire system became slower. But because the tail latency (p99) did not cross the failover threshold, the failover mechanism did not activate. As a result, they had to switch providers manually.

Why?
Because when the mean or p75 latency increases, it indicates a global slowdown. Responses are slower overall, and this impacts user experience.

Hence, I thought that I could use some of the methods I know to provide a solution. And here we are!!

Classical Methods

Fixed Threshold

if p75_latency > baseline_p75 × 2:
    Trigger failover.

Instead of looking only at the tail latency, i.e, the p99 or the exceptionally slow requests, we can look at the middle of the latency distribution too. Why? Because lower percentiles contain the global context, which we can simply compare to a threshold, and we will get a fundamental check for slow networks

Recent history

if p75_latency > Average p75 (last 10 min) × tau:
    Trigger failover.

Now, instead of using a fixed threshold, we can use the average of the latency scores in the past n minutes. This introduces a learned threshold instead of a fixed one. It is useful if we want to capture a sudden upshift in the latency. Tau can be used to give us a control over the value.

Detecting shape change in the histogram

if % of requests under 3s drops from 95% → 40%:
    Trigger failover.

Till now, we have tried looking at the percentile of request timings. But what if we focus on the distribution of request duration? i.e, instead of looking at ‘How much p75 or mean latency is shifting?’, we can ask ‘How many requests went from taking 2s to 3s or 4s?‘. If a large portion of requests move from “fast” to “slow” time ranges, we can detect a global latency shift.

Anomaly Detection Models

It is highly unlikely that there is a discussion of time series, and I don’t look for ML models that solve this. Hence, I tried to look for anomaly detection models for this use case.

We use ML models like Autoencoder, SVM, etc, to detect anomalies in time series. Though the initially mentioned problem does not require heavier ML models, we can still use these models to accurately analyze anomalous patterns in latency.

RNN-Based Detection

Recurrent Neural Networks can learn the expected temporal patterns of latency. If the observed latency deviates from the predicted pattern, the model flags an anomaly. These models struggle to maintain long-range dependencies, but are significantly faster than transformer-based ones.

Transformer Autoencoder-Based Detection

Transformer-based autoencoders learn a compressed representation of normal latency behavior. It can learn long-range dependencies better, but slower comparatively.

Conclusion

The current failover system mentioned in the blog typically detects isolated extreme delays, but global latency slowdowns show up in p50 and p75 latency, not p99. To maintain a responsive agentic system, monitoring should shift from single-call timeouts to distribution-aware metrics or learned anomaly models.

If you close your eyes, you can still tell where a sound is coming from: left, right, or behind you.
That’s because your brain compares the tiny time difference between when each ear hears the sound.

Now imagine trying to teach that same instinct to a machine, a radar tracking aircraft, a submarine detecting ships, or even your phone figuring out where your voice is coming from. That challenge is called Direction of Arrival (DoA) estimation: the art of teaching machines to locate where a signal comes from.

But just like humans developed from simple instincts to complex reasoning, DoA methods have evolved too.

Section 1: Classical Beamforming - The Ears of Early Machines

Before machines could reason, they had to learn to listen. The earliest DoA methods were built on physics and intuition, not statistics or AI.
Think of them as the ears of early machines, patient, simple, and surprisingly clever.

1. Delay-and-Sum Beamforming (DAS)

Imagine a row of microphones. The sound doesn’t hit them all at once. It reaches one, then the next, then the next, each a fraction of a moment apart.

If you delay the signals from the earlier microphones just enough so they all line up in time, and then sum them together, the sound from that direction gets amplified, while others fade away.
That’s Delay-and-Sum Beamforming, the most intuitive way to tell where a sound is coming from.

It’s simple, reliable, and easy to understand.

Strength: Simple, stable, and physically intuitive.
Limitation: Can’t separate sounds that are too close together.

2. Capon (MVDR) Beamforming

Capon’s method, also known as Minimum Variance Distortionless Response (MVDR), improves on DAS.

Instead of summing equally, it finds optimal weights for each sensor that:

• Keep signals from the desired direction unchanged,

• Minimize total output power (i.e., suppress everything else).

This forms a spatial filter that adapts to the environment similar to how your brain focuses on one voice in a noisy room.

Strength: High resolution, excellent at suppressing interference.
Limitation: Demands computation and accurate modeling.

3. Linear Prediction (LP) Method - Teaching Machines to Guess

Now imagine the machine not just listening but predicting.
The Linear Prediction method treats the signals across sensors like a melody: if it knows how one note sounds, it can guess the next.

By using patterns in the signals it’s already heard, it predicts what should come next and it forms a polynomial whose roots correspond to the directions of the sources.

It’s like a musician hearing a harmony and instantly knowing where each voice in the choir is standing.

Strength: Fast and effective, even when signals overlap.
Limitation: Needs careful tuning as a wrong assumption can create phantom sources.

Next, we move from the ears to the mind, where machines stop scanning blindly and start understanding patterns.

Section 2: Subspace-Based Methods - Seeing Between the Notes

After the early beamformers came a question.
Instead of just listening harder, scientists started asking, what if we could look inside the data itself? What if we could separate what’s meaningful from what’s just noise, the way a trained ear can pick out a violin’s voice in an orchestra?

That idea to find structure hidden within chaos gave birth to Subspace Methods.
They taught machines not just to listen, but to see between the notes.

1. The Core Concept - Signal vs. Noise Subspaces

Imagine all sensor readings arranged in a data matrix.
If multiple signals are arriving from different directions, this matrix contains overlapping patterns. Each direction contributes its own “signature.”

When we compute the covariance matrix of the array data, we can decompose it using something called Eigenvalue Decomposition (EVD).

The eigenvectors associated with large eigenvalues form the signal subspace. i.e: they represent real directions of arrival. The remaining eigenvectors (with small eigenvalues) form the noise subspace.

If we can isolate these subspaces, we can determine the directions without scanning every possibility blindly.

2. MUSIC (Multiple Signal Classification)

Its insight was beautifully simple: for the true signal directions, the “steering vector”, a kind of directional fingerprint should be perfectly orthogonal to the noise space.

So, MUSIC scans through angles and finds where this orthogonality holds, showing exactly where each sound originates.

It’s as if the machine listens to the world, and instead of hearing a blur, it hears distinct notes, each peak a voice in the crowd.

Strength: Very high resolution, even when sources are close together.
Limitation: Must know how many sources exist beforehand. It can’t play the tune without knowing how many instruments are in the band.

3. Root-MUSIC - Same Logic, Faster Computation

MUSIC requires scanning through all possible angles, which can be slow.
Root-MUSIC avoids this by turning the problem into a polynomial equation, then finding its roots directly the angles are derived from those roots.

Limitation: Works best with uniform linear arrays (ULAs)

4. ESPRIT: Estimation of Signal Parameters via Rotational Invariance

ESPRIT takes subspace methods one step further.
If your array has two identical subarrays, the signal seen by one is just a phase-shifted version of the other.

Mathematically, if the signal subspace of one subarray is S₁, and that of the other is S₂, then they’re related by:

S₂ = S₁ Φ

where Φ is a diagonal matrix encoding the phase shift, which depends on the direction of arrival.

By estimating Φ, ESPRIT directly computes the angles without any scanning or spectral plotting.

Strength: No search required, very fast
Limitation: Needs a specific array structure (e.g., two identical subarrays)

5. Unitary ESPRIT - Simplifying the Math

Unitary ESPRIT refines the original ESPRIT algorithm by converting complex-valued data into a real-valued form using unitary transformations.
This improves numerical stability and speeds up computation making it more practical for large systems.

Section 3: Maximum Likelihood Methods - The Search for the Most Probable Direction

The classical and subspace methods we’ve seen so far work well, but both rely on approximations.
They either scan for peaks in signal power or exploit mathematical separations between signal and noise spaces.

The Maximum Likelihood (ML) family takes a more rigorous approach.
Instead of scanning or separating, it asks a simple but powerful question:

“Given the data we actually measured, what is the most likely direction (or set of directions) that could have produced it?”

This transforms DoA estimation into a statistical optimization problem.

1. The Core ML Idea

Imagine your sensors picking up a mix of signals waves from multiple directions, tangled with noise.
ML treats this as a mystery to solve.

“if the sound truly came from this direction, how likely is it that I’d hear exactly this data?”

The direction that makes the data most probable wins.

This makes ML incredibly accurate, but also demanding. Like a perfectionist investigator, it wants all the data, all the time, and runs through every possibility before deciding.

2. Conditional ML vs. Unconditional ML

There are two main ways to set up the likelihood function, depending on how we treat the source signals (s).

Conditional ML (CML)

In this approach, we assume the signals themselves are deterministic but unknown. Solid, predictable sounds that the machine simply hasn’t seen yet.
ML then works out both the signals and the directions that best explain the recorded data.

Best for: strong, clear signals, like known reference tones in wireless systems.

Unconditional ML (UML)

The Unconditional approach is more humble; it assumes the world is messy.
Signals are random, influenced by uncertainty and noise.
So instead of trying to guess the exact signals, the algorithm integrates over all possible ones, focusing only on the most probable directions overall.

Best for: realistic, noisy environments where signals can’t be precisely modeled.

In both cases, the optimization involves searching over possible directions to maximize the likelihood but the mathematical form changes depending on this assumption.

3. Weighted Subspace Fitting (WSF) - A Practical Shortcut

While pure ML methods are theoretically optimal, they can be computationally demanding, especially when multiple sources are involved.

Weighted Subspace Fitting (WSF) is a clever compromise. It borrows the structure of subspace methods (like MUSIC and ESPRIT), but instead of full-blown likelihood optimization, it fits the model to the signal subspace using a weighted least-squares approach.

This preserves most of ML’s accuracy but avoids the full complexity of likelihood optimization.

4. Key Insight: ML as the “Gold Standard”

In theory, if we had infinite computing power and perfect models, ML would give the best possible accuracy for DoA estimation.

All other methods from MUSIC to sparse approaches can be viewed as approximations or simplifications of this principle.

But in practice, ML is often too slow or numerically unstable when data are limited or signals are weak.
That’s why researchers developed more efficient, structured approaches leading to modern sparse and Bayesian techniques.

Conclusion

From the humble delay-and-sum beamformer to the modern ML techniques, the field of Direction of Arrival (DoA) estimation has evolved from listening to reasoning.

Early methods treated the array like a set of synchronized ears: aligning, delaying, and summing signals to amplify what mattered.
Subspace approaches went further, uncovering the hidden geometry of the data, learning to tell real sources from random noise.
Maximum Likelihood introduced probability and optimization, turning the act of “hearing” into one of inference.

Together, these methods represent a clear evolution from analog intuition to algebraic structure, and finally to statistical intelligence.

Each generation brought sharper resolution, better noise resistance, and deeper insight into what the sensors were really “hearing.”

And each paved the way for what comes next, techniques that combine these foundations with new array geometries, tensor mathematics, and even deep learning.

Modern AI models are becoming increasingly larger and more sophisticated, but also increasingly expensive to operate. Every improvement seems to demand more memory, more GPUs, and more power. In the pursuit of trying to make better models, we might ask, Can we make AI think better without making it bigger?

The answer is fascinating. Instead of stacking more layers or adding more parameters, Mixture of Recursions(MoR) teaches a model to “think recursively”, to decide on the fly, how much effort each word deserves. Simple words get quick attention, while complex ones get deeper analysis. The result? Faster, leaner AI that performs similarly to their larger variants.

The problems with Big Models

Today’s AI models like GPT or Gemini are built on Transformers, a design that scales beautifully with size. And each boost in their intelligence comes with skyrocketing compute and memory costs. Training or even running these giants often demands specialized hardware, massive power, and corporate-scale budgets.

Recent research has tried two main tricks to make them lighter:

1. Parameter sharing - reusing the same layers multiple times instead of adding new ones.

2. Adaptive computation - letting the model decide when to stop thinking for “easy” parts.

The Mixture-of-Recursions paper steps in to fuse these two efficiency hacks into one elegant design.

The Idea Behind MoR

At its core, Mixture-of-Recursions (MoR) is about teaching an AI model to intelligently reuse its own layers. Instead of passing every word through a tall tower of layers, MoR uses a single shared block of layers again and again, like thinking in loops.

But the clever twist is that not every word needs the same amount of thinking. So MoR introduces a “router”, a lightweight decision-maker that decides how many times each word should loop through the block. Easy words might go through once. Tricky words like names, logic, or context-heavy phrases might go through several times.

This “mixture of recursions” lets the model think deeper only when needed. It saves computation, memory, and time, all while keeping performance on par with much larger models.

How It Works

The innovation lies in how MoR decides how many loops each token takes. A lightweight router analyzes token representations at each step and assigns a recursion depth per token, effectively deciding how much “thinking” it deserves.

Two routing styles mentioned are:

• Expert-choice routing - each recursion step (or “expert”) selects its respective tokens to compute further.

• Token-choice routing - each token chooses its full recursion path at the start.

Alongside routing, MoR refines KV caching, which determines how the model stores key-value pairs for attention. Instead of caching everything (as in standard Transformers), MoR uses:

• Recursion-wise caching, which stores only the active tokens’ KV pairs at each depth, reducing I/O and memory; or

• Recursive KV sharing, which reuses the first recursion’s cached keys and values across later loops, further cuts latency.

Together, these mechanisms let MoR dynamically allocate compute, reusing the same parameters while focusing attention and memory only where it matters most.

Results

Across model sizes from 135M to 1.7B parameters, MoR consistently establishes a new Pareto frontier, achieving lower validation loss and higher few-shot accuracy at the same or lower compute cost.

For example, with roughly half the parameters of a standard Transformer, MoR matched or surpassed it in benchmarks like ARC and MMLU. Under equal FLOPs, it processed more training tokens, thanks to adaptive routing that skips redundant computation.

In deployment, the benefits grow stronger. MoR’s continuous depth-wise batching, a batching method that reuses the shared block across tokens at different recursion depths, improves throughput by up to 2× (Figure 4a) compared to a vanilla model.

In essence, MoR combines the efficiency of weight-tying, the intelligence of adaptive computation, and the practicality of optimized caching, making it both faster and more resource-aware without sacrificing capability.

Conclusion

Mixture-of-Recursions represents a clear step toward smarter architectural efficiency. It shows that language models don’t always need more layers or parameters to improve; they need better control over when and where to think.

By combining recursive weight-sharing with adaptive routing and selective caching, MoR delivers high performance at a fraction of the compute cost. It’s not a radical reinvention of Transformers, but a refined evolution.

What’s ahead?

MoR changes how we think about scaling AI. Instead of endlessly building bigger models, it shows how to build smarter ones. Systems that can think deeply when needed and stay light when not.

For companies, this means cutting costs without cutting capability. For researchers, it opens a path to explore “thinking loops” that mimic how humans revisit tough problems instead of treating every thought equally.

In essence, MoR is a step toward AI that reasons more like us, adjusting its effort based on the difficulty of the task, not just the size of its brain.