NaN and Inf Detection
NaN and Inf counts are computed as standard tile statistics for every tensor Tensormorph indexes - not a special diagnostic mode you turn on, but always-available numbers that happen to be exactly what you need when a training run has gone wrong.
Why this needs to be structural, not a scan
A naive "scan every value looking for NaN" approach doesn't work at the scale Tensormorph targets - checking a trillion-parameter checkpoint element-by-element on demand would be far too slow to be useful interactively. Because NaN/Inf counts are computed once during indexing as part of the same mergeable tile-statistics pass as everything else, "does this model have any NaNs, and where" is answered by reading already-computed counts, aggregated up from tile to layer to model, not by re-scanning anything.
Finding the source of a corruption
Because the counts are mergeable, locating a NaN is a top-down narrowing process rather than a flat search:
- At the model level, the aggregate NaN/Inf count tells you immediately whether anything is wrong.
- Drilling into the Architecture view, layers and modules with a non-zero count are visually distinguishable (a dedicated overlay/shading option highlights them directly), so you can navigate straight to the affected region instead of reading through the whole tree.
- At the tile and scalar level, the Scalar Inspector shows the exact corrupted values.
Querying for corruption
The same counts are queryable through TQL - a query like "every tensor with a non-zero NaN count" runs as a filter over precomputed statistics across the whole model, immediately, rather than a live scan. This is also the basis for a breakpoint condition in runtime tracing: a capture point can be set to trigger specifically when a NaN first appears in a given tensor during a forward or backward pass, which is usually a far more useful place to stop than an arbitrary step number.
Related resources
- Tensor Statistics - where NaN/Inf counts sit among the other precomputed statistics.
- Runtime Traces - using NaN detection as a breakpoint condition during a live run.
- Outliers and Anomalies - for values that are finite but still suspicious.