This site is version 0.0.1. Most of the code on this site is drafted by an agent and needs review. I apologize for the AI-speak, we are working on rewriting it!
viv-lang.org
Python

vp.gpt.attention

The attention weights of every head of a GPT for one prompt.

Part of the distribution vipyr; it runs from a checkout of the repository. Planned on PyPI as pip install vipyr.

For each block and each head, the T by T matrix of softmax weights the attention takes its average with: row i says how much position i draws from positions 0 to i (the causal mask keeps every later position at zero). The weights are computed on the stream as it enters each block, by the same operations the forward pass runs, so they are the ones that produced the model's predictions.

Signature

vp.gpt.attention(model, prompt)

Parameters

model : vp.gpt.Model, required
A model from vp.gpt.train (r.model).
prompt : str | ndarray[int], required
One sequence of 1 to context tokens.

Returns

A frozen GptAttention. Iterating it gives the fields below in order; it also has params, tests, log and check().

weights : ndarray[float64]
Shape (layers, heads, T, T): row i of a head's matrix is how much position i takes from positions 0 to i (each row sums to 1).
labels : list[str]
The prompt's characters, for the axes (None for a model trained on token ids).

Raises

ValueError
Sizes that do not fit (width not a multiple of heads, a text no longer than the context), characters or token ids outside the model's vocabulary.
TypeError
An argument of the wrong type, or an unknown keyword.
vipyr.VivBuildError
An ImportError: no C compiler, or the program's C does not build.

See also

  • vp.gpt.train : Train a small GPT on a text, character by character, and get the model back as named arrays.
  • vp.gpt.generate : Continue a prompt with a trained GPT, one sampled token at a time.

Examples

import vipyr as vp
m = vp.gpt.train("hello world! " * 5, steps=30, seed=7).model
r = vp.gpt.attention(m, "hello wo")
print(r.weights.shape, r.weights[0, 0, -1].round(3))   # where the last position of head 0 looks
assert r.check()
(2, 2, 8, 8) [0.001 0.001 0.    0.    0.008 0.004 0.983 0.003]

Each example ran as written when these pages were generated, and the output shown is what it printed.

Checked against

Reference
Rows summing to 1 and zero above the diagonal (r.check()); the weights are those of the attention the forward pass runs (lib/nn's attention_weights, the same operations).

Source