Explore a run · lib/gpt/story_gpt.viv

A small GPT trained on one repeated line

A GPT with two blocks of two attention heads, 6,560 parameters, trained in your browser on the line "hello world!" repeated: token and position embeddings, RMSNorm, causal self-attention and a ReLU-squared MLP in each block, cross-entropy, the gradient from value_and_grad (the explorer shows its forward pass as the defs it was written in), and 30 steps of Muon for the block matrices and Adam for the rest. The loss falls from log 9 to 0.20, and from the prompt "hello" the model writes the line on.

Walk through the run: the program on one map, with a camera that follows each call and line as it runs.