Compiler internals
One program, every representation
This page follows one nine-line program through every form the compiler gives it, using
the compiler's own dumps — viv --emit tokens|ast|ir, the C
backend, the wasm emitter. A script regenerates all of it
(last run 2026-09-24).
The program
Sum of squares, 0 through 9. It is short enough to follow by hand and uses a definition, a call, a counted loop, mutation of a global, and output:
def sq(x) {
return x * x
}
total = 0.0
for i to 10 {
total = total + sq(i)
}
show total
Expected answer: 0² + 1² + … + 9² = 285. All three executors must
produce it.
- lexer.rscharacters → 40 tokens, every number literal an f64
- parser.rstokens → an AST with a byte span on every node
- lower.rsAST → typed, content-addressed IR in knotkit's store
- backend.rs / wasm_backend.rs / interp.rsthe C backend, the wasm backend and the interpreter
The lexer turns characters into tokens
Every numeric literal lexes as an f64: 10 lexes as
Num(10.0). The compiler makes a literal an Int later, when it
stands where the program expects one (an annotation, a parameter, a field). Newlines are real tokens, because
a newline is the statement terminator; inside parentheses or brackets the lexer suppresses
them so long expressions can wrap.
Def Ident("sq") LParen Ident("x") RParen LBrace Newline
Return Ident("x") Star Ident("x") Newline
RBrace Newline
Newline
Ident("total") Eq Num(0.0) Newline
For Ident("i") To Num(10.0) LBrace Newline
Ident("total") Eq Ident("total") Plus Ident("sq") LParen Ident("i") RParen Newline
RBrace Newline
Show Ident("total") Newline
Eof 40 tokens, regrouped one source line per row for reading. The dump itself is one token per line.
The parser turns tokens into an AST
Recursive descent, no backtracking; the first error stops the parse. The AST records
everything the source said, including a Span (a byte range) on every node,
which is what diagnostics and the workbench's red squiggles point at. The full dump is
219 lines. Here is def sq:
Program {
items: [
Def(
Def {
name: "sq",
name_span: Span {
source: SourceId(
0,
),
start: 4,
end: 6,
},
params: [
Param {
name: "x",
span: Span {
source: SourceId(
0,
),
start: 7,
end: 8,
},
ty: None,
},
],
ret: None,
body: [
Return {
value: Binary {
op: Mul,
lhs: Var {
name: "x",
span: Span {
source: SourceId(
0,
),
start: 23,
end: 24,
},
},
rhs: Var {
name: "x",
Excerpt. The AST has two separate Var("x") nodes for x * x;
the next stage merges them.
From the AST to content-addressed IR
Lowering type-checks the AST and builds knotkit's IR store, and every node's identity is a BLAKE3 hash of its kind, its payload, and its children's hashes — not a pointer, not an index. The dump prints that hash next to each node:
3dab37b2 Global { name: "i", ty: Num }
5d199bae Global { name: "total", ty: Num }
f282d0ce Func { name: "sq", params: [("x", Num)], ret: Num, locals: [], arena: false }
6e454fe1 Return
f19b3b33 Bin { op: Mul }
69a8093d Var { name: "x" }
69a8093d Var { name: "x" }
31428708 Main { locals: [], globals: [("i", Num), ("total", Num)] }
fd5be384 Assign { name: "total" }
cfcbeab1 Num(0.0)
6e4a5146 For { var: "i" }
38243316 Num(10.0)
deb8e613 Block
9ff783b5 Assign { name: "total" }
c9b80854 Bin { op: Add }
cc211d98 Var { name: "total" }
32863ce3 Call { callee: "sq" }
ad29d015 Var { name: "i" }
3ff5c94b Show { label: "total", is_arr: false }
cc211d98 Var { name: "total" }
Read the hashes. The two Var{x} operands of x * x are both
69a8093d — the AST had two nodes, the IR has one, referenced twice,
because they are structurally identical. Same for Var{total}
(cc211d98), shared between the loop body and the show. The tree
above has 20 positions and the store holds 18 distinct nodes.
The store keeps one entry per distinct hash, so identical subexpressions are shared without
a separate optimization pass, and results can be cached by hash. It is also how
viv hash and viv diff work: identical code has the same address.
Lowering is also where types are assigned (everything here is Num), where
assignments are hoisted into function-scoped locals, and where the typed vocabulary —
Func, Main, For, Bin, Show —
replaces the parser's untyped one. Past this point the AST is gone; no later phase sees it.
The C99 backend
knotkit's C backend walks the IR and emits portable C99. It is generic over the source
language: any language that implements the kit's code-generation interface gets a C99
backend. For this program the entire output is 24 lines, because the C runtime
functions a program uses are added only when it uses them, and this program uses one (viv_show_num):
#pragma STDC FP_CONTRACT OFF
#include <stdio.h>
static void viv_show_num(const char* label, double v) {
if (label[0]) printf("%s: ", label);
printf("%g\n", v + 0.0);
}
double i;
double total;
double sq(double x) {
return (x * x);
}
int main(void) {
total = 0.0;
for (i = 0.0; i < 10.0; i = (i + 1.0)) {
total = (total + sq(i));
}
viv_show_num("total", total);
return 0;
}
The C needs no library but libm and builds with cc sumsq.c -O2 -lm. Going through
C means clang applies the LLVM optimizer, and the Viv compiler contains no LLVM code.
$ viv sumsq.viv -o sumsq.c
wrote sumsq.c (18 IR nodes, 26 debug entries)
$ cc sumsq.c -O2 -lm -o sumsq && ./sumsq
total: 285 The program prints 285, and the compiler reports the 18 IR nodes from stage 3. The 26 debug entries map IR nodes to C lines; the workbench uses them to map a compiler diagnostic back onto your source.
The direct wasm backend
The second backend skips C entirely and emits WebAssembly bytecode by hand — no external crate, sections written byte by byte. It compiles from the same typed IR as the C backend, so it takes the whole language. The module does its own arithmetic, loops, calls and element access, and calls every other builtin in a runtime that lives inside the browser compiler and runs the interpreter's own code for it; that is how its output matches the interpreter's byte for byte. The program compiles to a module of 348 bytes:
$ viv sumsq.viv -o sumsq.wasm
wrote sumsq.wasm (348 bytes wasm) Those 348 bytes, section by section — parsed out of the module itself:
| Section | Bytes | Entries | Holds |
|---|---|---|---|
header | 8 | — | the wasm magic and version |
type | 30 | 6 | the distinct function signatures |
import | 94 | 6 | the runtime functions it calls, the host's flush, and the runtime's memory |
function | 5 | 2 | which signature each function uses |
global | 37 | 4 | the top-level bindings, then the addresses of the string table and a scratch pair |
export | 9 | 1 | the entry point the host calls |
data count | 3 | 2 | how many data segments there are, which memory.init needs to know first |
code | 150 | 2 | the function bodies |
data | 12 | 2 | the strings the program prints, and its constant tables |
| total | 348 | the whole module |
The compiled body of sq is six bytes,
20 00 20 00 a2 0f:
local.get 0 ; push x
local.get 0 ; push x again
f64.mul ; x * x
return To run the module, the host supplies its imports, the transcendentals and the output hooks:
$ node -e 'WebAssembly.instantiate(fs.readFileSync("sumsq.wasm"),
{ env: { exp: Math.exp, ..., show: x => console.log(x) } })
.then(m => m.instance.exports.run())'
total: 285 The interpreter
The third path is a tree-walking interpreter over the same parse. It lets a browser run Viv with no compile step, and a step budget stops a runaway loop before it hangs the tab. Every cell on this site runs on it. The interpreter runs the same program:
$ viv run sumsq.viv
total: 285 All three numbers above came from the same script run. The agreement is enforced: every feature goes into the interpreter and the C backend together, and the example suite checks that both print the same output. Agreement does not catch everything. An eigensolver doing single Jacobi rotations instead of full sweeps was wrong in both backends at once, and a physics program found it; an LU solve that interleaved row swaps with substitution was caught by its own residual check.
Where each stage lives
The language:
| Stage | File | Lines |
|---|---|---|
| Lexer | lang/viv/src/lexer.rs | 830 |
| Parser | lang/viv/src/parser.rs | 4,267 |
| AST | lang/viv/src/ast.rs | 553 |
| Lowering + C emission rules | lang/viv/src/lower.rs | 10,758 |
| Defunctionalization (higher-order → specialized) | lang/viv/src/defun.rs | 1,279 |
| Tree-walking interpreter | lang/viv/src/interp.rs | 6,970 |
| Direct wasm backend | lang/viv/src/wasm_backend.rs | 4,283 |
| Library surface + the wasm ABI | lang/viv/src/lib.rs | 2,565 |
| CLI | lang/viv/src/main.rs | 3,513 |
And the kit, shared with the sibling languages:
| Piece | File | Lines |
|---|---|---|
| content-addressed IR store | crates/knotkit/src/ir.rs | 184 |
| C backend + C AST + DebugMap | crates/knotkit/src/backend.rs | 1,494 |
| diagnostics as data | crates/knotkit/src/diag.rs | 161 |
| spans + source map | crates/knotkit/src/span.rs | 313 |
| injected I/O | crates/knotkit/src/io.rs | 90 |
37,260 lines in the files above, counted from the repository at build time.