This site is version 0.0.1. Most of the code on this site is drafted by an agent and needs review. I apologize for the AI-speak, we are working on rewriting it!
viv-lang.org
Python

vp.gpt.generate

Continue a prompt with a trained GPT, one sampled token at a time.

Part of the distribution vipyr; it runs from a checkout of the repository. Planned on PyPI as pip install vipyr.

Each new token is drawn from the softmax of the last position's logits divided by temperature, with the model seeing at most its last context tokens. temperature=1 samples the model as trained, lower values favor the likelier tokens, and temperature=0 takes the most likely token every time. The draws use Viv's rand() seeded by seed, so the same seed gives the same text.

Signature

vp.gpt.generate(model, prompt, *, n=100, temperature=1.0, seed=11)

Parameters

model : vp.gpt.Model, required
A model from vp.gpt.train (r.model).
prompt : str | ndarray[int], required
The text (or token ids) to continue.
n : int, default 100
How many tokens to add.
temperature : float, default 1.0
The logits are divided by it before the softmax; 0: the most likely token every time.
seed : int, default 11
The seed of the draws.

Returns

A frozen GptSample. Iterating it gives the fields below in order; it also has params, tests, log and check().

tokens : ndarray[int64]
The prompt's ids followed by the new ones.
text : str
The same as characters (None for a model trained on token ids).

Raises

ValueError
Sizes that do not fit (width not a multiple of heads, a text no longer than the context), characters or token ids outside the model's vocabulary.
TypeError
An argument of the wrong type, or an unknown keyword.
vipyr.VivBuildError
An ImportError: no C compiler, or the program's C does not build.

See also

  • vp.gpt.train : Train a small GPT on a text, character by character, and get the model back as named arrays.
  • vp.gpt.loss : The mean next-token cross-entropy of a GPT on a text.

Examples

import vipyr as vp
m = vp.gpt.train("hello world! " * 5, steps=30, seed=7).model
r = vp.gpt.generate(m, "hello", n=24, seed=11)
print(r.text)
assert r.check()
hello world! hello world! hel

Each example ran as written when these pages were generated, and the output shown is what it printed.

Checked against

Reference
The sample of study/compare/ml/tiny_gpt.py from the same model and seed, to the character; at temperature 1 the draws of lib/gpt's sample (lib/gpt/api.viv). python/tests/test_gpt.py.

Source