Skip to content
Docs
Core Concepts
Dtype and quantization

Dtype and Quantization

A tensor's dtype is one of the fixed metadata fields covered in Shape, Rank, Stride and Layout; this page focuses on the part of the dtype space that needs its own explanation - quantized dtypes, where the stored value isn't the real value.

Ordinary vs. quantized dtypes

An ordinary floating-point dtype (float32, bfloat16, float16) stores something close enough to the real value directly - reading the number is using the number. A quantized dtype (int8, int4, and lower-bit packed formats) stores an integer that only approximates the real value once combined with extra parameters - a scale, and often a zero-point. Tensormorph's dtype metadata always distinguishes the two, and the Scalar Inspector shows both the raw stored integer and the dequantized value for any quantized scalar, rather than picking one to display.

💡

Dtype is orthogonal to shape and stride - two tensors with identical shape can differ only in dtype, and a single quantization pass typically leaves shape and stride completely untouched while changing dtype and adding scale/zero-point metadata.

Granularity

Quantization parameters (scale and zero-point) can be shared at different granularities:

  • Per-tensor - one scale/zero-point pair for the whole tensor.
  • Per-channel - one pair per output channel, common for weight quantization since different channels often have meaningfully different value ranges.
  • Per-group - one pair per small run of adjacent values, common for aggressive low-bit weight-only quantization.

Finer granularity generally preserves more accuracy at the cost of more metadata; Tensormorph's quantization-boundary overlay (see the Architecture view) draws a line wherever this granularity changes, so a per-group scheme's boundaries are visible directly on the tensor.

Where this shows up elsewhere

  • Quantized Models - the applied, format-level view of everything on this page.
  • Quantization Diff - comparing a quantized tensor against its full-precision source.
  • Sparsity - aggressive quantization can round small values to exactly zero as a side effect, which looks like pruning-induced sparsity but isn't.

Related resources

Shape, rank, stride and layoutShards and storage