8c9a374Unbiased Monte-Carlo entropy estimation for quantized filtered Gaussian seriesJeremy Magland 1# timeseries-entropy
3Unbiased Monte-Carlo estimation of the entropy of a quantized filtered
4Gaussian time series:
6 x iid N(0, sigma^2) -> y = h * x -> z = round(y)
8The estimand is the conditional entropy H(z_{M+1} | z_1..z_M) in bits, which
9decreases to the entropy rate of z — the true lossless compression limit in
10bits/sample — as the past window M grows beyond the memory of the process.
12Companion to [timeseries-compressibility](https://github.com/concept-collection/timeseries-compressibility).
14## Method
161. **Stationary conditional sampling.** Draw x from the prior and push it
17 through the pipeline to get a past z_1..z_M. The generating x is an exact
18 draw from p(x | z), so a Gibbs chain started there is already in
19 stationarity — no burn-in bias. Each Gibbs conditional is a box-truncated
20 normal; after each sweep the free tail latent is drawn fresh, emitting one
21 exact sample of z_{M+1}. The samples form a stationary, autocorrelated
22 discrete chain.
232. **Unbiased entropy of the chain's marginal.** Plug-in entropies of blocks
24 whose sizes double per level are combined by Rhee–Glynn randomized
25 telescoping with antithetic half-block corrections
26 Delta_m = h(B_m) - [h(B_m^1) + h(B_m^2)]/2, truncated at a random level N
27 with P(N >= m) = 2^(-r m) and reweighted. The expectation is exactly
28 H(z_{M+1} | that past) despite the plug-in bias at every finite block size
29 and despite the autocorrelation, which affects only the variance.
303. **Average over independent pasts** to get H(z_{M+1} | z_1..z_M) with a
31 valid standard error. The only remaining approximation to the entropy rate
32 is the finite window M.
34## Install
36 pip install -e .
38Requires numpy and scipy.
40## Usage
42```python
43from timeseries_entropy import estimate_conditional_entropy, kernels
45est = estimate_conditional_entropy(kernels.moving_average(8), sigma=4.0)
46print(est.mean, est.se) # bits/sample, over independent pasts
47```
49Lower level: `ConditionalChain(kernel, sigma, past, rng).draw(k)` yields the
50stationary chain of z_{M+1} samples, and `unbiased_entropy(draw, n0, r, rng)`
51is one randomized-telescoping realization for any stationary discrete chain.
53## CLI
55 timeseries-entropy --sigma 4 --filter moving-average --width 8
56 timeseries-entropy --sigma 8 --filter lowpass --high 3000 --rate 30000
57 timeseries-entropy --sigma 2 --filter none --pasts 8
59Filters match the web app: `none`, `moving-average`, `lowpass`, `bandpass`,
60`first-difference`.
62## Tuning
64- `r` (default 1.5) sets the truncation tail P(N >= m) = 2^(-r m). Finite
65 expected work needs r > 1; finite variance needs E[Delta_m^2] to decay
66 faster than 2^(-r m). Run `--pilot 6` to see the RMS Delta_m decay before
67 trusting a value.
68- `n0` (default 128) is the base block size; `thin` inserts extra Gibbs
69 sweeps per emitted sample to cut autocorrelation for slowly mixing
70 (narrowband, large-sigma) settings.
71- `--past` sets M; increase it until the estimate stops moving to approach
72 the rate.