Sparsity

🚧

The zero-count statistic and its role in Tensormorph's rendering are confirmed design facts; the specific sparsity-pattern classification described below (structured vs. unstructured) is standard domain vocabulary applied to this product, not sourced from a local reference document - there is no local PyTorch torch.sparse API reference in this workspace to adapt from.

Sparsity - the fraction of a tensor's values that are exactly zero - is measured directly from the zero count already computed as part of every tensor's tile statistics, so checking how sparse a tensor is never requires a separate pass over its data.

Why sparsity is worth checking

A tensor's sparsity ratio alone is a useful signal for several different situations that all produce zeros for different reasons:

Pruning - a model that's had unimportant weights zeroed out deliberately, where sparsity is the point.
Quantization artifacts - aggressive low-bit quantization can round small values to exactly zero as a side effect, which looks the same as pruning in a zero-count but has a different cause.
MoE routing - an expert that's rarely or never selected by the router will show near-total sparsity in its accumulated contribution, distinct from sparsity in the expert's own weights.

Because these look identical in a raw zero-count, distinguishing them usually means cross-referencing sparsity against outliers and anomalies or against a quantization diff, rather than reading the zero-count in isolation.

Structured vs. unstructured sparsity

Two tensors with the same overall sparsity ratio can look completely different once you look at where the zeros are:

Unstructured sparsity - zeros scattered without a regular pattern. Visually, this shows up as noise in the Matrix/Heatmap view's sparsity-mask shading mode.
Structured sparsity - zeros following a regular pattern (whole rows, columns, or fixed-size blocks zeroed together, or an N:M pattern such as 2:4). Structured sparsity is what most sparse-compute hardware paths can actually accelerate; unstructured sparsity generally can't be exploited for a speedup even at a high ratio.

The sparsity-mask shading mode is what makes this distinction visible at a glance - a structured pattern reads as a visible grid or stripe pattern in the heatmap, where unstructured sparsity reads as texture with no discernible regularity.

Related resources

Tensor Statistics - the zero-count statistic sparsity is measured from.
Matrix and Heatmap Editor - where the sparsity-mask shading mode lives.
Quantization Diff - for distinguishing quantization-induced zeros from deliberate pruning.

Histograms and distributions Outliers and anomalies