This site is version 0.0.1. Most of the code on this site is drafted by an agent and needs review. I apologize for the AI-speak, we are working on rewriting it!
viv-lang.org

Machine learning

Transformers

demoAIMachine learningwritten with an agent#7ee1c1d37f
Adam Zettel@adam

A transformer trained with Muon

One attention head, 336 parameters, a vocabulary of six. The task is to output the previous token, which a transformer can do only by learning to look one position back. It is trained by Muon, an optimizer that orthogonalizes the momentum with five Newton-Schulz steps before stepping along it. Chance is a loss of 1.792; sixty steps bring it to 0.00058 with every position right. In the attention matrix it ends with, each position puts its weight on the one before it.

Training loss, from chance (log 6) to 0.00058
what it gotwhat guessing gets
10⁻³10⁻²10⁻¹10⁰0102030405060training stepmean cross-entropy
What the head learned to look at
position asking
position attended to
6 × 6 grid
✓ 8 tests passchecked when the site was built#transformers#muon#attention#optimization
ReadOpen in the workbench

Look inside this program

  • Atlas

    The program’s map: every function and which calls which.

  • Explore

    Run it and step through what each part did, with the code, values and plots beside it.

Optimizers

articleAIMachine learningwritten with an agent#7aac6fa51e
Adam Zettel@adam5 min read

Muon orthogonalizes the update

Muon is an optimizer that keeps the directions of a gradient matrix and gives every one of them a step of about the same size, however lopsided the gradient is. Five Newton-Schulz steps get close to that without an SVD. On a problem whose inputs differ in size a hundredfold, it ends at least 4.5 times closer to the answer than SGD, momentum or Adam at their best step sizes, and behind momentum in training loss.

Five Newton-Schulz steps on a gradient10⁻⁴10⁻²10⁰012345Newton-Schulz stepeach singular value
the eight singular valuesthe band, 0.68 to 1.14
Distance from the true map, each optimizer at its best step size
SGDSGD with momentumAdamMuon
10⁰10¹10²020406080100step|W - W*|

24 paragraphs · 89 functions · 5 min

✓ 10 tests passchecked when the site was built#optimization#muon#newton-schulz#linear-algebra
Read the articleOpen in the workbench

Look inside this program

  • Atlas

    The program’s map: every function and which calls which.

  • There’s no recorded run of this program yet.