Skip to content
Docs
Inspect and Analyze
Expert similarity

Expert Similarity

🚧
The MoE Expert Galaxy view and its use of a similarity-driven layout are confirmed design facts; the specific similarity metrics described below are standard techniques applied to that view, not sourced from a local reference document.

In a Mixture-of-Experts layer, two experts can end up nearly identical after training (redundant capacity that isn't earning its parameter cost) or sharply divergent (genuine specialization). Telling these apart is what expert-similarity analysis is for, and it's the computation that drives the default spatial layout of the MoE Expert Galaxy view.

Computing similarity

Like SVD, PCA and effective rank, pairwise expert similarity is an explicit analysis - it requires comparing actual weight data across every pair of experts in a layer, not just combining precomputed tile statistics. Requesting it for a layer computes a similarity score for every expert pair (typically a cosine similarity or correlation between flattened weight vectors, or between each expert's dominant principal directions), producing a full pairwise similarity matrix for the layer's experts.

Reading the result

The similarity matrix is viewable directly as a small heatmap (one row/column per expert), but its main use is driving the Expert Galaxy's idle-state layout: experts position themselves in 3D space so that similar experts cluster together and dissimilar ones sit apart, turning an n-by-n similarity matrix into something you can read spatially at a glance rather than by scanning a grid of numbers.

What high similarity suggests

Two experts with very high similarity are worth double-checking against a few different explanations, since the fix differs by cause:

  • Under-trained routing - the router rarely sends distinct-enough inputs to both, so neither has had the chance to specialize away from the other.
  • Genuine redundancy - both experts converged to a similar function because the task didn't need the extra capacity, which is a legitimate pruning candidate.
  • Initialization proximity - early in training, or for a freshly-initialized model, experts can be similar simply because they haven't diverged yet.

Cross-referencing a routing trace from Runtime Traces - how often each expert is actually selected - is usually what separates "under-trained" from "genuinely redundant."

Related resources

SVD, PCA and effective rankWeight diff