Skip to content
Docs
Working With Models
Mixture-of-Experts models

Mixture-of-Experts Models

Mixture-of-Experts (MoE) architectures route each token to a small subset of a much larger set of expert modules, rather than passing every token through every parameter. Tensormorph treats this as a first-class case rather than rendering an MoE layer as an ordinary stack of modules with a gate - routing has its own dedicated 3D view, and its own analysis tools, because "which experts actually fire, and how similar are they to each other" is usually the question that matters most for this architecture family.

The MoE Expert Galaxy view

The MoE Expert Galaxy is one of Tensormorph's eight 3D views (alongside Architecture, Layer Stack, Tensor Volume, Parameter Map, Diff, Morph, and Runtime Flow). It lays experts out as spatially arranged nodes rather than a flat list, and renders routing weight as flow between them:

  • Idle view - experts positioned by similarity (see Expert Similarity below), so structurally related experts cluster together without needing to inspect each one individually.
  • With a captured routing trace - routing weight renders as flow between the gate and the experts it activated; selecting a single token highlights that token's actual path through the layer, distinguishing it from the aggregate flow of every other token in the same batch.

Because rendering here follows the same render-statistics, not-parameters principle as every other view, the flow you see is derived from captured routing decisions and tile-level statistics - never a re-execution of the model to guess at routing behavior.

Recognizing MoE structure on import

Tensormorph's ModelAdapter layer recognizes common MoE module patterns (a gate/router producing per-expert weights, alongside a set of structurally parallel expert sub-modules) and marks them accordingly in the Semantic Model Graph, so the Expert Galaxy view is available immediately on open rather than requiring manual annotation. An architecture the adapter doesn't recognize still opens normally through the fallback adapter - you lose the specialized MoE views for that layer, not the ability to inspect it.

Expert similarity

Two experts in the same layer can end up highly similar after training (redundant capacity) or highly divergent (genuine specialization); telling these apart is a core MoE-specific analysis. See Expert Similarity for how Tensormorph computes and visualizes pairwise expert similarity, which is also what drives the Expert Galaxy's default spatial layout.

Related resources

Quantized modelsMultimodal models