Shards and Storage
Shape, Rank, Stride and Layout covers a tensor whose storage lives in one place. This page covers what happens once storage is split across files or devices - the foundation Sharded Checkpoints builds on.
Two independent kinds of splitting
- File-level sharding - a checkpoint's tensors spread across multiple files purely for manageable file size, with an index mapping each tensor name to its file. Every tensor still has one complete, unsplit storage; only its file location is split.
- Placement sharding - a single logical tensor's storage genuinely divided across devices, described by a device mesh (the logical grid of ranks/devices) and a placement per mesh dimension:
- Shard - each device holds a distinct slice.
- Replicate - every device holds the full tensor.
- Partial - each device holds a partial value awaiting a reduction across devices.
Local vs. full views
A placement-sharded tensor has both a local view (only the data resident on one device) and a full view (the reassembled logical whole). Tensormorph's views default to showing the full, seamless tensor without necessarily materializing it entirely on one device - reassembly happens as part of rendering, not as a separate manual step.
Requesting the full view of a very large placement-sharded tensor is a real cost, gated by the same exactness-never-silently-degrades principle as any other expensive request - Tensormorph tells you when it's substituting an approximation rather than doing so silently.
Why storage splitting doesn't affect diffing
A weight diff always compares full, reassembled logical tensors - never raw per-shard or per-file bytes - specifically so that a change in how a checkpoint is split (a different shard count, a different device mesh) never shows up as a spurious difference between two checkpoints that are otherwise numerically identical.
Related resources
- Sharded Checkpoints - the format-level, applied continuation of this page.
- Level of Detail and Streaming - how partial/lazy loading interacts with sharded storage.
- GPU Backend Compatibility - reassembling a sharded tensor across a heterogeneous set of backends.