Multimodal Models

A multimodal model - vision-and-text, audio-and-text, or more towers still - usually isn't one uniform stack of layers; it's several sub-networks (a vision tower, a text tower, an audio encoder) that meet at one or more fusion points. Tensormorph doesn't require a special mode to open one, but a few things behave differently once a model has more than one input modality.

Multiple towers, one address space

Every recognized architecture is read into the Semantic Model Graph by a ModelAdapter written for that architecture; a multimodal model's adapter is what tells Tensormorph "these layers are the vision tower, these are the text tower, this module is where they fuse." In the Outliner and the Architecture view, each tower appears as its own top-level branch of the model, with fusion modules positioned where they structurally sit - so you can navigate "the vision tower" as a coherent unit rather than an unlabeled prefix of a flat layer list.

If Tensormorph has no dedicated adapter for a given multimodal architecture, the fallback adapter still opens it - every tensor is inspectable, the address space still applies - but layers aren't automatically grouped by modality; you're navigating a flat structure rather than tower-aware one.

Shape semantics differ by modality

The same rank-3 or rank-4 tensor shape means something different depending on which tower it belongs to - a sequence-of-tokens axis in a text tower is not the same kind of axis as a patch-grid axis in a vision tower, even when the numbers happen to match. This is why shape alone, without knowing which tower a tensor belongs to, is not enough to safely align two multimodal checkpoints for comparison: Comparing Two Model Revisions needs tower-aware alignment for a multimodal pair, not just shape-matching.

Inspecting a fusion point

A fusion module - where a vision tower's output and a text tower's embeddings combine - is often the most informative place to inspect in a multimodal model, since it's where cross-modal behavior actually happens. It shows up in the Architecture view like any other module; the Inspector panel and Scalar Inspector work on it identically to a single-tower model, since neither depends on which tower a tensor came from.

Related resources

• Model, Architecture, Layer and Module - how the ModelAdapter/fallback-adapter split builds the address space this page assumes.
• Comparing Two Model Revisions - alignment considerations specific to multi-tower models.
• Mixture-of-Experts Models - another case where architecture-aware recognition unlocks dedicated views beyond the generic fallback.