Chapter 13 — Distributed Training
Supported topology matters
Replicated data parallelism, where every rank holds a full model copy, is the supported multi-GPU route described in Revision 2. Do not assume that a sharded, partitioned, or tensor-parallel model is equivalent: IRIS™ needs the model state required by the particular diagnostic.
Use the distribution's integration recipe for your framework and topology. Check the selected writer and actual coverage in the run's log. Distributed training does not make every diagnostic globally calibrated; some readings are explicitly local or uncalibrated, and should be described that way.
Compiled models
torch.compile support depends on PyTorch version and whether the model
uses fullgraph=True. With fullgraph, per-step motion or training-affinity
channels can be absent even when representation captures are available.
Older PyTorch versions have additional probe-capture limits. Follow the
version-specific notes in your installed manual and test the exact compiled
wrapper before a long job.
Lifecycle and preemption
Keep the training script's existing signal, checkpoint, and requeue behavior. The Bench is a separate viewer; stopping or restarting it does not stop training. On preemption, inspect the saved bundle and the last recorded frame rather than assuming that every scheduled observation landed. On resume, check the run identity and step sequence. A duplicate extraction at an already measured step does not create a new measurement.
Honest coverage
Look for explicit absent, local, uncalibrated, or skipped states in the reported diagnostics. A local curvature or gradient reading is not a measurement of the full distributed objective simply because the trainer used several devices. Preserve those qualifiers in reports and comparisons.
Measure overhead on your own hardware and topology. An instrumentation cost from a different model, training-step duration, probe set, or accelerator does not predict this run's cost.
