This site is version 0.0.1. Most of the code on this site is drafted by an agent and needs review. I apologize for the AI-speak, we are working on rewriting it!
viv-lang.org

vp.gguf.read

Read a GGUF file's metadata and tensor table.

Part of the distribution vipyr; it runs from a checkout of the repository. Planned on PyPI as pip install vipyr.

vp.gguf.read reads the head of the file (the first megabyte, doubled until the tensor table fits) and hands the bytes to file_index of lib/gguf/api.viv, which parses the header, the metadata and the tensor table as docs/gguf.md lays them out and returns them as flat numbers; Python turns those into a dict and a list. The tensor data is not read.

f = vp.gguf.read("stories15M-q8_0.gguf")
f.metadata["llama.embedding_length"]      # 288
f.metadata["tokenizer.ggml.tokens"][:3]   # ['<unk>', '<s>', '</s>']
f.tensors[0]                              # TensorInfo(name='token_embd.weight', shape=(32000, 288), ...)
f.tensor_info("blk.0.attn_q.weight")      # its row of the table, by name

The result goes to vp.gguf.tensor(path, name, file=f) so that each tensor does not read the table again.

Signature

vp.gguf.read(path)

Parameters

path : str | Path, required
The GGUF file.

Returns

A frozen GgufFile. Iterating it gives the fields below in order; it also has params, tests, log and check().

path : str
The file read.
metadata : dict
Key -> int, float, bool, str or list (an array; an array of arrays is a list of lists), in the file's order.
tensors : list[TensorInfo]
The tensor table in the file's order: name, shape (NumPy's order, the reverse of GGUF's dims), type (ggml's name), offset from the file's start, n_bytes.
alignment : int
General.alignment, or 32 when the file does not set it.
data_offset : int
Where the data section starts: the end of the table rounded up to the alignment.

Raises

VivStop
A file that is not GGUF (it does not start with the bytes GGUF), a version other than 3, a value type GGUF does not define, a string that is not UTF-8, or a table that runs past the end of the file.
TypeError
An argument of the wrong type, or an unknown keyword.
vipyr.VivBuildError
An ImportError: no C compiler, or the program's C does not build.

See also

Notes

A GGUF file is a header (the bytes GGUF, version 3, the number of tensors and of metadata pairs), the metadata (typed key-value pairs: eight integer types, two float types, bool, UTF-8 strings, and arrays of any of them, arrays included), the tensor table (name, dimensions, ggml type, offset into the data section), padding to general.alignment (32 when the file does not set it), and the data. A float32 value comes back as the Python float equal to it; integers of every width as Python ints.

References

  1. The GGUF file format specification, ggml's docs/gguf.md, at commit 3612c0af93f2cfc8f048295300a61187723b12ae (lib/gguf/UPSTREAM.md). https://github.com/ggml-org/ggml/blob/master/docs/gguf.md

Examples

import os
import vipyr as vp
r = vp.gguf.read(os.path.join(vp.viv_root(), "lib/gguf/tests/stories260K.gguf"))
print(r.metadata["general.architecture"], r.metadata["llama.block_count"], len(r.metadata["tokenizer.ggml.tokens"]))
print(r.tensor_info("blk.0.attn_q.weight"))
assert r.check()
llama 5 512
TensorInfo(name='blk.0.attn_q.weight', shape=(64, 64), type='F32', offset=276576, n_bytes=16384)

Each example ran as written when these pages were generated, and the output shown is what it printed.

Checked against

Reference
gguf-py 0.19.0's GGUFReader: the metadata and the tensor table of lib/gguf/tests/tiny.gguf (every value type) and of llama2.c's stories260K, stories15M-q8_0 and stories15M-q4_0, printed the same way and identical byte for byte in the interpreter, as C and as wasm (tools/gguf/check.sh); python/tests/test_gguf.py.

Credits

  • The GGUF reader is a rewrite, from the GGUF specification (docs/gguf.md), of the reader in ggml's src/gguf.cpp and gguf-py's gguf_reader.py; the dequantizers are a rewrite of dequantize_row_* in ggml's src/ggml-quants.c, with the block structs of src/ggml-common.h (ggml at commit 3612c0af93f2, MIT, The ggml authors), and are tested number for number against gguf-py 0.19.0. The test model is llama2.c's stories260K (MIT).

Source