This site is version 0.0.1. Most of the code on this site is drafted by an agent and needs review. I apologize for the AI-speak, we are working on rewriting it!
viv-lang.org

vp.gguf.tensor

One tensor of a GGUF file, dequantized to float32.

Part of the distribution vipyr; it runs from a checkout of the repository. Planned on PyPI as pip install vipyr.

vp.gguf.tensor finds the tensor in the table, reads its bytes, and dequantizes them with dequant of lib/gguf/api.viv (lib/gguf/quants.viv): the result is a float32 array in NumPy's order (GGUF lists dimensions fastest first; the shape here is the reverse).

The quantized types store blocks of weights with a scale per block:

| type | weights per block | bytes per block | a weight is | |---|---|---|---| | Q8_0 | 32 | 34 | a signed byte times the block's half-precision scale | | Q4_0 | 32 | 18 | (a 4-bit q minus 8) times the scale | | Q4_1 | 32 | 20 | a 4-bit q times the scale plus the block's minimum | | Q4_K | 256 | 144 | 4 bits with 6-bit scales and minimums for each 32 | | Q6_K | 256 | 210 | 6 bits (4 low, 2 high) with a signed 8-bit scale for each 16 |

F32, F16 and BF16 are converted exactly. ggml computes Q4_1's sum in single precision, and so does lib/gguf (fround), so the values are gguf-py's bit for bit.

Signature

vp.gguf.tensor(path, name, *, file=None)

Parameters

path : str | Path, required
The GGUF file.
name : str, required
The tensor's name (blk.0.attn_q.weight).
file : GgufFile, default None
The result of vp.gguf.read(path), to reuse its table; None reads it.

Returns

A frozen GgufTensor. Iterating it gives the fields below in order; it also has params, tests, log and check().

array : ndarray[float32]
The tensor's values in its shape (NumPy's order), float32 as ggml computes in; every quantized value is exact in float32.
info : TensorInfo
Its row of the tensor table.

Raises

KeyError
No tensor of that name.
NotImplementedError
A type lib/gguf does not read (the IQ types, Q2_K, Q3_K, Q5_0, Q5_1, Q5_K, ...).
TypeError
An argument of the wrong type, or an unknown keyword.
vipyr.VivBuildError
An ImportError: no C compiler, or the program's C does not build.

See also

References

  1. ggml's dequantize_row_* functions, src/ggml-quants.c, at commit 3612c0af93f2cfc8f048295300a61187723b12ae; gguf-py 0.19.0's gguf/quants.py, the reference the values are checked against. https://github.com/ggml-org/ggml/blob/master/src/ggml-quants.c

Examples

import os
import vipyr as vp
path = os.path.join(vp.viv_root(), "lib/gguf/tests/stories260K.gguf")
r = vp.gguf.tensor(path, "blk.0.attn_q.weight")
print(r.info.type, r.array.shape, r.array.dtype, r.array[0, :3])
assert r.check()
F32 (64, 64) float32 [ 0.04439279  0.07974771 -0.00892535]

Each example ran as written when these pages were generated, and the output shown is what it printed.

Checked against

Reference
gguf-py 0.19.0's gguf.quants.dequantize: every tensor of lib/gguf/tests/tiny.gguf (one of each type lib/gguf reads) and of stories260K, stories15M-q8_0 and stories15M-q4_0, number for number as binary32 bit patterns, in the interpreter, as C and as wasm (tools/gguf/check.sh); lib/gguf/tests/tiny.ref holds gguf-py's values for tiny.gguf (python/tests/test_gguf.py).

Credits

  • The GGUF reader is a rewrite, from the GGUF specification (docs/gguf.md), of the reader in ggml's src/gguf.cpp and gguf-py's gguf_reader.py; the dequantizers are a rewrite of dequantize_row_* in ggml's src/ggml-quants.c, with the block structs of src/ggml-common.h (ggml at commit 3612c0af93f2, MIT, The ggml authors), and are tested number for number against gguf-py 0.19.0. The test model is llama2.c's stories260K (MIT).

Source