Copyright © 2026 Saccadic LLC · Patent pending.

Browse chapters
  1. Release NotesChanges by version
  2. Chapter 1What IRIS Is and Is Not
  3. Chapter 2The IRIS Data Model
  4. Chapter 3How IRIS Measures
  5. Chapter 4Safe Interpretation and the First 15 Minutes
  6. Chapter 5Dashboard Overview
  7. Chapter 6Installation
  8. Chapter 7Configuration
  9. Chapter 8Build the Model Spec and Probe Set
  10. Chapter 9Minimal Tracker Integration
  11. Chapter 10Launch the Bench and Verify Capture
  12. Chapter 11Runtime Control, Signals, Metrics, and Logs
  13. Chapter 12Integration Patterns by Model Family
  14. Chapter 13Distributed Training
  15. Chapter 14Artifacts, Reports, Export, and Security
  16. Chapter 15Troubleshooting
  17. Chapter 16Whole Model and Layer Structure
  18. Chapter 17Domain Training and Affinity
  19. Chapter 18Attention-Head Diagnostics
  20. Chapter 19MoE Expert and Router Diagnostics
  21. Chapter 20Facets and Emergent Pathways
  22. Chapter 21Gradient Residuals, Stability, Curvature, and Update Efficiency
  23. Chapter 22Adaptive Cadence and Event Detection
  24. Chapter 23CLI Reference
  25. Chapter 24Configuration-Key Reference
  26. Chapter 25API Reference
  27. Chapter 26File and Schema Reference
  28. Chapter 27Glossary and Concept Index
  29. Chapter 28The Flow Map
  30. Appendix ADiagnostic Availability Matrix
  31. Appendix BLLM Prompt Library
On this page

Chapter 13 — Distributed Training

Supported topology matters

Replicated data parallelism, where every rank holds a full model copy, is the supported multi-GPU route described in Revision 2. Do not assume that a sharded, partitioned, or tensor-parallel model is equivalent: IRIS™ needs the model state required by the particular diagnostic.

Use the distribution's integration recipe for your framework and topology. Check the selected writer and actual coverage in the run's log. Distributed training does not make every diagnostic globally calibrated; some readings are explicitly local or uncalibrated, and should be described that way.

Compiled models

torch.compile support depends on PyTorch version and whether the model uses fullgraph=True. With fullgraph, per-step motion or training-affinity channels can be absent even when representation captures are available. Older PyTorch versions have additional probe-capture limits. Follow the version-specific notes in your installed manual and test the exact compiled wrapper before a long job.

Lifecycle and preemption

Keep the training script's existing signal, checkpoint, and requeue behavior. The Bench is a separate viewer; stopping or restarting it does not stop training. On preemption, inspect the saved bundle and the last recorded frame rather than assuming that every scheduled observation landed. On resume, check the run identity and step sequence. A duplicate extraction at an already measured step does not create a new measurement.

Honest coverage

Look for explicit absent, local, uncalibrated, or skipped states in the reported diagnostics. A local curvature or gradient reading is not a measurement of the full distributed objective simply because the trainer used several devices. Preserve those qualifiers in reports and comparisons.

Measure overhead on your own hardware and topology. An instrumentation cost from a different model, training-step duration, probe set, or accelerator does not predict this run's cost.