Skip to content
Tensormorph
Profiling
Runtime traces

Runtime Traces

Everything covered so far in this documentation works on a static checkpoint. Runtime tracing is how Tensormorph looks at a model while it's actually running - capturing activations, gradients, and timing from a live forward or backward pass, structurally similar to how a debugger attaches to a running program.

RuntimeProbe: where a trace comes from

A RuntimeProbe is anything that can supply live data to Tensormorph. Three kinds are confirmed:

  • A bundled local runner (tensormorph-probe) - attaches via forward hooks on a PyTorch model running in the same process or a connected local process.
  • A serving-framework plugin - for models running under vLLM or SGLang, where Tensormorph observes an already-running inference server rather than launching its own.
  • .tmrun replay - a previously captured trace, replayed after the fact with no live process involved at all. See File Formats.

Every RuntimeProbe exposes the same interface regardless of source: describe what it can capture, subscribe to a stream of events, request an on-demand capture, and set a breakpoint - so a capture configuration built against a live local run and one replayed from a .tmrun file behave identically from Tensormorph's side.

Breakpoints and capture points

A breakpoint attaches to a specific node in the Semantic Model Graph - a module, a tensor - the same address every other Tensormorph view already navigates by. Its trigger condition is a TQL predicate evaluated against live statistics, most commonly something like tensor.nan_count > 0 to stop the instant a NaN first appears (see NaN and Inf Detection), rather than an arbitrary step number.

Probe levels: trading depth for overhead

Capturing everything, always, isn't free - deeper capture changes the performance of the run being observed. Tensormorph exposes four probe levels (0 through 3), each capturing progressively more at progressively higher overhead; which level is appropriate depends on whether you're doing a quick sanity check or a deep investigation of one specific module.

⚠️

Dropped events - captures skipped because a probe fell behind the run, or because a level didn't request them - always render as visible gaps in the timeline, never interpolated or silently filled in. A gap means "we don't know," not "nothing happened here."

Reading a trace

A captured trace scrubs like the checkpoint-index playhead in Signal and Distribution Editor, but along a time axis instead of a static ordered one - stepping through captured events, with every other view (Architecture, Matrix/Heatmap, Scalar Inspector) updating to show that moment's captured data for whatever's currently selected.

Related resources

Export transformed modelsActivations