This site is version 0.0.1. Most of the code on this site is drafted by an agent and needs review. I apologize for the AI-speak, we are working on rewriting it!
viv-lang.org
Python

vp.gpt.loss

The mean next-token cross-entropy of a GPT on a text.

Part of the distribution vipyr; it runs from a checkout of the repository. Planned on PyPI as pip install vipyr.

The loss is the mean, over every window of context tokens of the text (the windows training draws its batches from) and every position in the window, of minus the log probability the model gives the token that comes next, in nats. A model that knows nothing scores log(vocab).

Signature

vp.gpt.loss(model, tokens)

Parameters

model : vp.gpt.Model, required
A model from vp.gpt.train (r.model), its arrays as they are now.
tokens : str | ndarray[int], required
Text (or token ids) longer than the model's context.

Returns

A frozen GptLoss. Iterating it gives the fields below in order; it also has params, tests, log and check().

loss : float
The mean next-token cross-entropy (nats) over every window of context tokens.

Raises

ValueError
Sizes that do not fit (width not a multiple of heads, a text no longer than the context), characters or token ids outside the model's vocabulary.
TypeError
An argument of the wrong type, or an unknown keyword.
vipyr.VivBuildError
An ImportError: no C compiler, or the program's C does not build.

See also

  • vp.gpt.train : Train a small GPT on a text, character by character, and get the model back as named arrays.
  • vp.gpt.generate : Continue a prompt with a trained GPT, one sampled token at a time.

Examples

import vipyr as vp
text = "hello world! " * 5
m = vp.gpt.train(text, steps=30, seed=7).model
print(vp.gpt.loss(m, text).loss)
m.blocks[0].mlp.up[:] = 0.0                      # the arrays are the model: edit one and measure again
r = vp.gpt.loss(m, text)
print(r.loss)
assert r.check()
0.16581612858387473
0.6677118658423731

Each example ran as written when these pages were generated, and the output shown is what it printed.

Checked against

Reference
The loss of the PyTorch reimplementation (python/tests/gpt_reference.py) on the same windows, within 10−12 relative (python/tests/test_gpt.py).

Source