Why LPC+ANS gets 14x when the entropy rate says 23x#
Setting: --sigma 5 --filter bandpass --low 300 --high 2000 --taps 31 --rate 30000.
All numbers are bits/sample on the quantized signal z (ratio = 16 / bits).
Verdict#
The entropy estimate is right, and the gap is real. LPC+ANS is not suboptimal by accident — integer-residual prediction followed by a memoryless entropy coder structurally cannot reach the conditional entropy of this process. A codec that codes each sample against a discretized Gaussian centred at the real-valued prediction reaches 22.3x with a verified lossless round-trip; a Monte-Carlo model-based codec reaches ~23.7x.
The mechanism#
After optimal linear prediction from the quantized past, the prediction error of y_next has std s ~= 0.28 quantization steps. The conditional distribution of z_next = round(y_next) is therefore a Gaussian bump covering ~1-3 integer bins, and its shape depends strongly on the fractional part of the prediction mu: mu near a bin centre gives ~0.4 bits, mu near a bin edge gives ~1.1 bits.
- An ideal conditional coder uses mu per sample: mean entropy ~= 0.72 bits.
- LPC+ANS codes the integer residual z - floor-prediction with one pooled histogram: that histogram is the mixture over all fractional parts, entropy ~= 1.07 bits (semi-analytic; 1.10 measured with the app's 15-bit integer coefficients).
The ~0.35-0.38 bit gap is the 14x-vs-22x discrepancy. Raising the LPC order does nothing (14.6x plateau by order 64): the loss is in the residual coding, not the prediction.
Measurements (4M-sample realizations; codecs charged for all side info)#
| method | bits/sample | ratio |
|---|---|---|
| raw int16 | 16 | 1x |
| order-0 entropy of z | 1.954 | 8.2x |
| LPC(32) + ANS (app's method, reproduced) | 1.101 (+~1.5% ANS) | 14.3x |
| LPC(128) + ANS ceiling | 1.098 | 14.6x |
| conditional-Gaussian arithmetic codec (real bytes, exact round-trip) | 0.7184 | 22.27x |
| Gibbs posterior-predictive codec (achievable rate, 60 sweeps) | 0.674 +/- 0.065 | 23.7x |
| timeseries-entropy estimator, M=512 (default settings) | 0.615 +/- 0.065 | 26x |
| timeseries-entropy estimator, M=1024, thin=2 | 0.658 +/- 0.046 | 24.3x |
| semi-analytic conditional entropy (rounding-as-noise model) | 0.723 | 22.1x |
The true entropy rate is bracketed: <= 0.718 is proven by real compressed
bytes that round-trip; ~0.674 is achievable by the model-based codec; the
Monte-Carlo estimator's 0.62-0.68 is consistent with both. The estimator's
Rhee-Glynn pilot (--pilot 6) shows RMS Delta_m <= 0.016 decaying at
exponent ~2, so its r=1.5 truncation is sound at these settings.
The codec that proves it (codec_gaussian.py)#
- Fit an order-64 real-coefficient predictor on the block (Levinson); send coefficients as float32 in the header (270 B total).
- Fit a single residual scale s (grid search on the block); send it too.
- For each sample: mu_t = dot(coeffs, previous 64 decoded samples); code z_t with an arithmetic coder under P(k) proportional to Phi((k+1/2-mu_t)/s) - Phi((k-1/2-mu_t)/s), tails folded into edge bins, 16-bit frequencies. Decoder computes the identical mu_t from decoded samples, so tables agree bit for bit.
- First 64 samples are coded under the marginal N(0, std_z) table.
1,048,576 samples -> 94,157 bytes = 0.7184 bits/sample = 22.27x,
np.array_equal(z, decoded) == True. Encode/decode ~7 s each in pure
Python+numpy.
gibbs_predictive_rate.py pushes further: both sides could run the
timeseries-entropy Gibbs sampler on the decoded past (shared seed) and code
against the Rao-Blackwellized posterior predictive of the generative model —
the nonlinear optimum. Its measured cross-entropy is 0.674 +/- 0.065
bits/sample (23.7x). The -log2-of-a-mean is Jensen-biased upward, so more
sweeps would only lower it toward the true conditional entropy.
Files#
analysis_spectral.py— semi-analytic model: Kolmogorov innovation variance of S_z = sigma^2|H|^2 + 1/12; conditional vs mixture entropy; empirical Wiener-predictor rates by order.lpc_ans_repro.py— exact port of the app's integer LPC (15-bit coeffs, shared shift, floor prediction, int16 wrap); reproduces 14.3x.codec_gaussian.py— the round-tripping conditional-Gaussian codec.gibbs_predictive_rate.py— achievable rate of the Monte-Carlo model-based codec, reusing timeseries_entropy.ConditionalChain.entropy_cli_run.log,entropy_pilot.log,codec_gaussian_1M.log,gibbs_predictive.log— raw outputs.
Notes for the app / estimator#
- The app's hollow "order-0 entropy of the residual" bar is the LPC+ANS ceiling, and the measured 14x sits right on it — the coder is fine; the prefilter family is the limit.
- The tenth method now exists in the app:
src/compress/conditionalGaussian.ts, computed with the other nine in the compression worker and drawn as its own "conditional" chart group. In-browser at these settings, 100k block: LPC(32)+ANS 1.119 bits (14.30x) vs cond. Gaussian AC 0.738 bits (21.68x). - With prediction-error std s, the fractional-phase loss of integer-residual coding is ~E[H(mix)] - E[H(cond)]; it vanishes for s >> 1 (white/large sigma) and grows as s drops below ~0.5 — worst exactly in the narrowband, moderate-sigma regime this parameter set sits in.