`vp.stats.describe`

The mean, variance, skewness and kurtosis of a sample, with sums that keep their digits.

Part of the distribution vipyr; it runs from a checkout of the repository. Planned on PyPI as pip install vipyr.

vp.stats.describe(x) returns what scipy.stats.describe returns: the count, the smallest and largest value, the mean, the variance (divided by n - 1 unless ddof says otherwise), and the biased skewness g1 and excess kurtosis g2.

ddof is NumPy's "delta degrees of freedom": the variance divides by $n - ddof$ , so 1 gives the sample variance and 0 the population variance.

method picks how the variance is computed. method="two_pass" (the default) is the corrected two-pass algorithm of Chan, Golub and LeVeque:

s^{2} = \frac{\sum _{i} d _{i}^{2} - \frac{1}{n} ( \sum _{i} d _{i} ) ^{2}}{n - ddof}, d_{i} = x_{i} - x ˉ,

with every sum in NumPy's pairwise order. With exact arithmetic the deviations sum to zero and the second term vanishes; with a rounded mean they do not, and subtracting the term removes the error the rounding left. method="welford" updates the mean and the sum of squared deviations one value at a time (Welford 1962), for data seen once. Neither computes $\sum x_{i}^{2} - n x ˉ^{2}$ , which loses every digit of a sample like $1 0^{9} + (4, 7, 13, 16)$ .

Signature

vp.stats.describe(x, *, ddof=1, method='two_pass')

Parameters

x : ndarray[float64], required: The sample, any shape (flattened).
ddof : int, default 1: The variance divides by n - ddof.
method : str, default 'two_pass': The variance slot: 'two_pass' (corrected two-pass, pairwise sums) or 'welford' (one pass).

Returns

A frozen DescribeResult. Iterating it gives the fields below in order; it also has params, tests, log and check().

nobs : int: The number of values.
minmax : tuple: The smallest and largest value.
mean : float: The mean.
variance : float: The variance, divided by n - ddof.
skewness : float: G1, the biased sample skewness (SciPy's skew default).
kurtosis : float: G2, the biased excess kurtosis (SciPy's kurtosis default).

Raises

ValueError: Fewer than two values, or a method that is not registered.
TypeError: An argument of the wrong type, or an unknown keyword.
vipyr.VivBuildError: An ImportError: no C compiler, or the program's C does not build.

Examples

import numpy as np
import vipyr as vp
x = 1e9 + np.array([4.0, 7.0, 13.0, 16.0])
r = vp.stats.describe(x)
print(r.mean, r.variance)                  # 1000000010.0 30.0
assert r.check()

1000000010.0 30.0

Each example ran as written when these pages were generated, and the output shown is what it printed.

Checked against

Reference: lib/stats/describe.viv's tests; python/tests/test_stats.py against NumPy and SciPy, and on 10⁹ + x samples where the textbook formula sum x^2 - n mean^2 loses every digit.

Source

Viv: lib/stats/describe.viv
Viv: lib/stats/api.viv