Daisythe teaching edition of IRISBeta Exit
Runs in your browsernon-interactive · loops
{{ eyebrow }}

{{ title }}

{{ subtitle }}

{{ sg.t }}

{{ gradTerm }}{{ gs.t }}

{{ tipText }}
{{ calcNote }} {{ bestHead }} {{ bestBody }}
Optimizer{{ optStatus }}
Turn every slider the way its arrow points, by the learning rate times the arrow’s length. Then calculate again.
{{ lrTxt }}
Loss {{ loss }}

The optimizer gives up

This single neuron cannot learn both rules across these examples.

It can match one addition/subtraction pair, but not all the pairs here. Adding needs a second-number weight of +1; subtracting needs −1. We need one neuron for each rule.

Neuron 1

{{ compactFirst }}{{ compactOperator }}{{ compactSecond }}={{ truthTxt }}
{{ compactFirst }}·w1+{{ compactOperator }}·w2+{{ compactSecond }}·w3=y
{{ compactFirst }}·{{ compactWeightFirst }}+{{ compactOperatorEmbedding }}·{{ compactWeightOperator }}+{{ compactSecond }}·{{ compactWeightSecond }}={{ answerTxt }}
Loss {{ loss }}

Neuron 1

{{ compactFirst }}{{ compactOperator }}{{ compactSecond }}={{ truthTxt }}
{{ compactFirst }}·w1+{{ compactOperator }}·w2+{{ compactSecond }}·w3+b=y
{{ compactFirst }}·{{ compactWeightFirst }}+{{ compactOperatorEmbedding }}·{{ compactWeightOperator }}+{{ compactSecond }}·{{ compactWeightSecond }}+{{ compactBias }}={{ compactAnswer }}
Loss {{ loss }}

{{ ln.name }}

Output {{ ln.output }}ReLU {{ ln.reluState }}
Answer {{ answerTxt }} Goal {{ truthTxt }}

Good job! {{ lessonSolvedText }}

{{ optNote }}

{{ optHalt }}

Prompt & context

{{ dg.eqTile }}{{ tr.t }}Drop a tile here

Embedding

{{ tr.e }}

The AI

AI{{ dg.dots }} {{ dg.status }}

{{ dg.name }} {{ dg.big }}
Loss {{ loss }}{{ allRightTxt }}close enoughfrom a fresh start: {{ freshErr }}
Need a hint? The optimizer follows the gradient.Move all four sliders first
Need a hint? The optimizer follows the gradient.Move all four sliders first
Equationsclick one to follow it
Loss{{ loss }} from a fresh start{{ freshErr }} {{ lightsKey }}
{{ stackTop }} {{ stackBottom }}
tokens
{{ t.w }}
Embeddings
{{ t.code }}
{{ n.name }} {{ n.state }} RIP
{{ ln.label }} {{ ln.math }}
{{ al.label }}{{ al.math }}
{{ answerTxt }}the answer
should be{{ truthTxt }} error{{ errTxt }} within half a point
New: not trained yet {{ fe.text }}
Already trained {{ fe.text }}
{{ fn.name }}
weights
× {{ k.val }}
New: not trained yet {{ fe.val }}
Already trained {{ fe.val }}
Loss{{ loss }}{{ allRightTxt }} close enough from a fresh start: {{ freshErr }}
{{ stackTop }} {{ stackBottom }}
Neuron 1 one token per rowPrompt Embeddings weightsrow by row{{ weightsHead }}row by row {{ tb.t0 }}{{ tb.t0 }} {{ tb.e0 }} × {{ tb.w0 }}{{ ks.a.val }} = {{ tb.p0 }} {{ tb.t1 }}{{ tb.t1 }} {{ tb.e1 }} × {{ tb.w1 }}{{ ks.b.val }} = {{ tb.p1 }} {{ tb.t2 }}{{ tb.t2 }} {{ tb.e2 }} × {{ tb.w2 }}{{ ks.c.val }} = {{ tb.p2 }} bias + {{ tb.w3 }}{{ ks.d.val }} = {{ tb.p3 }} {{ tb.t0 }}{{ tb.t0 }} {{ tb.e0 }} ×{{ tb.w0 }}={{ tb.p0 }} {{ tb.t1 }}{{ tb.t1 }} {{ tb.e1 }} ×{{ tb.w1 }}={{ tb.p1 }} {{ tb.t2 }}{{ tb.t2 }} {{ tb.e2 }} ×{{ tb.w2 }}={{ tb.p2 }} bias +{{ tb.w3 }}={{ tb.p3 }} sum {{ tb.total }} {{ tb.a }}A, the output {{ tb.correct }}correct {{ tb.err }}error sum {{ tb.total }} {{ ruleWord }}{{ tb.ruleWhy }} {{ tb.a }}A, the output {{ tb.correct }}correct {{ tb.err }}error

Each curve is one slice through the loss surface. How sharply a slice bends is its curvature. The Hessian collects those bends for every direction you could slice.

{{ hessSurf.xLabel }}
{{ hessSurf.yLabel }}

{{ explainHead }}

{{ eg.t }}
Working it out
{{ sa.copy1Txt }} {{ sa.copy2Txt }}
{{ sa.sumTxt }}

Each embedding is multiplied by its weight, and the results are added up. That total is the neuron’s output.
Football nicknames
Math
Drag a tile into the Prompt, or click it.
Prompt Contexteach box is one token Embeddings Output {{ rw.t }} {{ rw.c }} {{ rw.e }} AI {{ dg.name }} {{ dg.era }} {{ dg.big }} {{ dg.dots }}{{ dg.status }} {{ dg.eqTile }} {{ dg.hist }} {{ dg.trick }}
Look closely: {{ dg.postitText }}
Prompt
{{ tkl.promptPlain }} {{ mp.t }}
Context
{{ mc.t }}
AI{{ tkl.aiDots }} {{ tkl.aiStatus }}
Output
{{ tkl.outTxt }}

↶ Each new output joins the context for the next pass.

Prompt
{{ twd.t }} {{ tkl.promptPlain }}
Context
{{ cx.t }}
AI {{ tkl.aiDots }}{{ tkl.aiStatus }}
Output {{ tkl.outTxt }}
Prompt → Tokens
Seektounderstand.
The full stop is its own token.
Context
Seektounderstand.Beforebeingunderstood.
AI
Output . Each new token is fed back into the context.
Prompt Embeddings {{ tr.t }} {{ tr.c }}
Prompt{{ tb.t0 }}{{ tb.t1 }}{{ tb.t2 }} Neuron 1 gives{{ n21.part }}+Neuron 2 gives{{ n22.part }}={{ tb2.ans }}{{ tb2.ans }}A, the output{{ tb.correct }}correct{{ tb.err }}error Neuron 1 one token per rowPrompt Embeddings weightsrow by row {{ tb.t0 }}{{ tb.t0 }} {{ tb.e0 }} ×{{ ks.a.val }}={{ tb.p0 }} {{ tb.t1 }}{{ tb.t1 }} {{ tb.e1 }} ×{{ ks.b.val }}={{ tb.p1 }} {{ tb.t2 }}{{ tb.t2 }} {{ tb.e2 }} ×{{ ks.c.val }}={{ tb.p2 }} bias +{{ ks.d.val }}={{ tb.p3 }} {{ g2.sumWord }} {{ tb.total }} ReLU of the total{{ n21.why }}{{ n21.out }} how much{{ vk1.label2 }}{{ n21.o }}×{{ vk1.val }}={{ n21.part }} same embeddingssame
embeddings Neuron 2 {{ tb.e0 }} ×{{ ks2.a.val }}={{ tb2.p0 }} {{ tb.e1 }} ×{{ ks2.b.val }}={{ tb2.p1 }} {{ tb.e2 }} ×{{ ks2.c.val }}={{ tb2.p2 }} bias +{{ ks2.d.val }}={{ tb2.p3 }} {{ g2.sumWord }} {{ tb2.total }} ReLU of the total{{ n22.why }}{{ n22.out }} how much{{ vk2.label2 }}{{ n22.o }}×{{ vk2.val }}={{ n22.part }} {{ g2.ansLabel }} {{ tb2.ans }} {{ tb2.ans }}A, the output {{ tb.correct }}correct {{ tb.err }}error
Optimizer{{ optStatus }}
Turn every slider the way its arrow points, by the learning rate times the arrow’s length. Then calculate again.
{{ lrTxt }}
{{ optNote }} {{ optHalt }}

{{ bg.t }}
{{ solvedHead }}{{ solvedLine }}{{ solvedTerm.a }}{{ solvedTerm.b }}{{ solvedTerm.c }}
{{ endNoteText }}
Look closely: {{ postit }}
{{ hr.name }}
{{ k.label }}
{{ k.val }}
{{ arrowKey }}
{{ nextLabel }}

{{ popupHead }}

{{ popupLine }}