# Runtime Traces

Everything covered so far in this documentation works on a static checkpoint. Runtime tracing is how Tensormorph looks at a model while it's actually running - capturing activations, gradients, and timing from a live forward or backward pass, structurally similar to how a debugger attaches to a running program.

## RuntimeProbe: where a trace comes from

A **`RuntimeProbe`** is anything that can supply live data to Tensormorph. Three kinds are confirmed:

-   **A bundled local runner** (`tensormorph-probe`) - attaches via forward hooks on a PyTorch model running in the same process or a connected local process.
-   **A serving-framework plugin** - for models running under vLLM or SGLang, where Tensormorph observes an already-running inference server rather than launching its own.
-   **`.tmrun` replay** - a previously captured trace, replayed after the fact with no live process involved at all. See [File Formats](/docs/reference/file-formats).

Every `RuntimeProbe` exposes the same interface regardless of source: describe what it can capture, subscribe to a stream of events, request an on-demand capture, and set a breakpoint - so a capture configuration built against a live local run and one replayed from a `.tmrun` file behave identically from Tensormorph's side.

## Breakpoints and capture points

A **breakpoint** attaches to a specific node in the [Semantic Model Graph](/docs/core-concepts/model-architecture-layer-module) - a module, a tensor - the same address every other Tensormorph view already navigates by. Its trigger condition is a [TQL](/docs/reference/expressions-and-queries) predicate evaluated against live statistics, most commonly something like `tensor.nan_count > 0` to stop the instant a NaN first appears (see [NaN and Inf Detection](/docs/inspect-and-analyze/nan-and-inf-detection#querying-for-corruption)), rather than an arbitrary step number.

## Probe levels: trading depth for overhead

Capturing everything, always, isn't free - deeper capture changes the performance of the run being observed. Tensormorph exposes four probe levels (0 through 3), each capturing progressively more at progressively higher overhead; which level is appropriate depends on whether you're doing a quick sanity check or a deep investigation of one specific module.

Dropped events - captures skipped because a probe fell behind the run, or because a level didn't request them - always render as visible gaps in the timeline, never interpolated or silently filled in. A gap means "we don't know," not "nothing happened here."

## Reading a trace

A captured trace scrubs like the checkpoint-index playhead in [Signal and Distribution Editor](/docs/editors/signal-and-distribution-editor), but along a time axis instead of a static ordered one - stepping through captured events, with every other view (Architecture, Matrix/Heatmap, Scalar Inspector) updating to show that moment's captured data for whatever's currently selected.

## Related resources

-   [Activations](/docs/profiling/activations) and [Gradients](/docs/profiling/gradients) - the two most common things a runtime trace captures.
-   [TQL: Expressions and Queries](/docs/reference/expressions-and-queries) - writing a breakpoint condition.
-   [File Formats](/docs/reference/file-formats) - the `.tmrun` format a trace is saved as.