Chapter 8 — Build the Model Spec and Probe Set
Creating a Spec Dict/File
{
"name": "my-model",
"model_type": "transformer",
"architecture": "decoder-only",
"n_layers": 6,
"n_heads": 8,
"d_model": 512,
"ff_dim": 2048,
"vocab_size": 50257,
"max_seq_len": 1024,
"normalization": "pre_ln",
"activation": "gelu",
"has_bias": true
}
For MoE models, add:
{
"moe": {
"n_experts": 8,
"n_experts_per_tok": 2,
"router_type": "top_k"
}
}
Valid enum values
model_type:transformer,gnn,ssm,cnn,rnn,hybrid,declared. Usedeclaredonly when you are supplying a measurement adapter instead of letting IRIS™ walk the module tree — see "Declared architectures" in Chapter 12."transformer"requiresn_heads >= 1, andd_modelmust be divisible byn_heads— IRIS needsn_headsto split attention outputs into per-head slices for per-head k95 and head-role diagnostics; a missing or zeron_heads, or one that doesn't divided_model, raisesIrisConfigErrorat construction, before any GPU work.architecture: free string; common valuesdecoder-only,encoder-only,encoder-decodernormalization:pre_ln,post_ln,rmsnorm,none(usenonefor architectures without LayerNorm, e.g. Nanda et al. 2023)activation:gelu,gelu_new,swiglu,relu,silu,tanhframework(optional):pytorch,jax,tensorflow
Mixed-bias models
has_bias is the global default. For models where attention and MLP
disagree on bias usage, use the per-layer overrides:
has_bias_attn— bias on Q/K/V/O projections (null = inherit)has_bias_mlp— bias on MLP linears (null = inherit)
Example for a Nanda-style model whose attention is bias-free but MLP
has both b_in and b_out:
{"has_bias": true, "has_bias_attn": false}
Creating a Probes File
A probe is a concrete model input with a domain label and, optionally, a subdomain and expected answer. Use inputs representative of the work you want to inspect. The probe set is fixed inspection material, not a random held-out test of generalization.
For character tokenization, every character in the input and expected
answer must be present in your model's char2id. Match the training
tokenizer's actual IDs and special tokens. This single-probe schema example
shows the structure; it is not a sufficient evaluation set:
{
"tokenizer": {
"type": "char",
"char2id": {"0":3,"1":4,"2":5,"3":6,"4":7,"5":8,"6":9,"7":10,"8":11,"9":12," ":13,"+":14,"=":15,"\n":16},
"pad_id": 0, "bos_id": 1, "eos_id": 2
},
"terminator": "\n",
"probes": [
{"input":"17 + 25 = ","expected_answer":"42","domain":"addition"}
]
}
Replace that example vocabulary with the model's real mapping before use. Out-of-vocabulary probes can be dropped; review warnings and the resulting probe coverage rather than assuming every input was accepted.
Tokenizers and answer spans
For a Hugging Face tokenizer, pass tokenizer=your_hf_tokenizer and
probe_texts= to the constructor rather than using probes_path.
A custom callable tokenizer returns (ids, mask) and can optionally return
an answer_mask as a third element. Use the integration reference supplied
with your distribution for model-specific details.
When a probe is a question/answer pair, put the question and separator in
input and declare the completion in expected_answer. Answer-span
readings score the completion positions; EOS, when available, is included
as the answer's end. For character data without EOS, a declared terminator
can provide that end. Whole-text token accuracy is a different reading
and is not substituted for an absent answer-span measurement.
Coverage and labels
Use multiple domains when asking comparative domain questions. Single-domain runs cannot support comparisons between different domains; affected views either report that limit or explicitly show pooled data. Use subdomains when a finer category is useful. A probe's category is its subdomain when declared, otherwise its domain.
Increase probe diversity and examine stability across your own tests. More probes also increase measurement work. Wrong answers still produce useful activations to inspect, but a pattern on a small fixed probe set does not establish task-wide generalization or a circuit's identity.
