Chapter 25 — API Reference
Core lifecycle
from iris.inline import IrisTracker
tracker = IrisTracker(
output_dir="/path/to/run",
spec_dict=my_spec,
probes_path="my_model.probes.json",
)
output_dir identifies the run root. spec_dict describes the actual
architecture. The probe/tokenizer route must match the model: a probes file
for the supported file workflow, or tokenizer/probe-text arguments for the
supported callable/Hugging Face workflow. Model-specific constructor options
are documented with your installed distribution.
| Interface | Contract |
|---|---|
should_extract(step) |
Poll every optimizer step and extract when it returns true. Legacy every_n_steps is ignored. |
extract(model, step=step) |
Capture the model at that training step. Repeating an already measured step does not write another frame. |
flush() |
Finish pending analysis before exiting. |
set_optimizer(optimizer) |
Register the optimizer that actually steps; enables supported training-dynamics readings. |
set_loss_fn(fn) |
Supply a compatible scalar-loss callback before the first extraction. |
set_eval_batch_fn(fn) |
Supply a fixed evaluation batch callback before the first extraction. |
set_validation_set(val_x, val_y) |
Optionally register validation inputs and targets for supported telemetry. |
Trainer-derived metrics
report_loss(loss, step=None) accepts a scalar loss, including a supported
scalar tensor. Pass step= when exact freshness matters.
report_metrics(step=None, *, ...) accepts independent fields:
| Loss | Accuracy |
|---|---|
train_loss_overall |
train_acc_overall |
train_loss_by_domain |
train_acc_by_domain |
holdout_loss_overall |
holdout_acc_overall |
holdout_loss_by_domain |
holdout_acc_by_domain |
Pass only values your code actually measured. For this interface, loss is token-weighted mean cross-entropy over non-padding targets; accuracy is top-1 next-token accuracy on the same target set, from 0 to 1. Per-domain fields use domain-name mappings. Use a clearly defined UDS for a metric with a different meaning.
tracker.report_metrics(
step=step,
holdout_loss_overall=validation_token_cross_entropy,
holdout_acc_overall=validation_next_token_accuracy,
)
Absent fields stay absent. Probe metrics are not trainer keyword fields: IRIS™ measures them on its fixed probes. Do not label them as independently sampled held-out accuracy or silently substitute whole-text accuracy for answer-span accuracy.
User Defined Signals
report_signal(index, value, label=..., unit=..., description=...) records
a scalar with a stable identity and definition. Non-finite values remain
gaps rather than valid plotted values; unsupported vectors or tensors do
not become a scalar series. Keep the signal's provenance explicit when
comparing it with IRIS-derived measurements.
Callback and integration limits
Keep measurement callbacks side-effect-free and independent of the training iterator. Register callbacks before the first extraction, inspect custom hooks, and validate a short run. Diagnostic availability depends on the model, captures, callbacks, topology, and resources, not merely on a method having been called. Chapter 21 explains the curvature and residual qualifications.
