Paper outline#

Title#

The entropy rate of a noisy quantized signal and how far lossless codecs sit from it, with application to extracellular voltage recordings

Thesis#

The entropy rate of this signal class is computable. Here is where real codecs sit relative to it.

Scope decisions. The unbiased Monte-Carlo estimator is the contribution; the analytic formula is a fast approximation validated against it. The paper is descriptive — why codecs fall short (the gap decomposition) is a separate paper. Target ~10–12 pages: estimator construction and the unbiasedness statement in the body, proof in an appendix.

The estimator earns trust three ways: it is provably unbiased; it agrees with the theoretically justified analytic formula; and advanced codecs approach it from above.

---

1. Introduction#

• Instruments emit quantized samples of a noisy continuous process. Lossless compression is a real cost, and the practice is empirical — try codecs, keep the winner. Nobody knows how much is left.

1.1 Prior work#

Organized around what each body of work supplies and what it leaves open.

• Ephys compression benchmarks. Buccino et al. benchmarked general-purpose and audio codecs on large-scale Neuropixels data, finding the audio codecs (WavPack, FLAC) outperform general-purpose compressors and that lossy modes buy substantial further ratio without measurably harming spike sorting. It is the reference point for this paper — and it is codec-versus-codec. With no entropy-rate estimate, "best" can only mean best of those tried, and the distance to what is achievable is unknown. Same for the surrounding literature on ephys and scientific-array compression (Blosc, Zarr/HDF5 filter pipelines).
• Lossless audio coding. FLAC, Shorten, WavPack, MPEG-4 ALS — the predict-then-code-the-residual architecture the ephys benchmarks inherit, and its Rice/Golomb residual coding.
• Entropy-rate estimation. Plug-in block entropy and its downward bias; Lempel–Ziv and compression-based estimators. The last are upper bounds of unknown tightness, so they cannot serve as the reference against which compressors are judged — using a compressor to estimate the limit that a compressor is being measured against begs the question.
• Unbiased MCMC. Rhee–Glynn randomized truncation; Jacob et al. — the machinery §3 builds on.
• Quantization theory. Bennett's dither model, Gray's quantization theory, high-resolution limits; the source of the 1/12 correction in §4.

1.2 This paper#

• The missing quantity is the entropy rate $\bar H$. For this class it is not analytically available (rounding destroys Gaussianity) and not reliably estimable by plug-in methods (wide alphabet, long memory).
• We give an unbiased estimator of $\bar H$, an analytic approximation that agrees with it, and a benchmark of standard codecs against it — on synthetic sources with an exact limit, and on real extracellular recordings, both raw wideband and bandpass-filtered.
• Headline numbers to preview.
• Contributions.

2. Setup#

• $z_t = \operatorname{round}(y_t)$, $y$ stationary Gaussian with PSD $S(f)$, unit quantization step. Amplitude in quantization steps; $S$ says everything.
• $y = h * x$ is the spectral-factorization form — $h$ is the minimum-phase factor of $S$, not a filter anyone applied.
• Estimand: $\bar H = \lim_{M\to\infty} H(z_{M+1} \mid z_1..z_M)$.
• Accounting: every reported size is the full encoded byte count, includes side information, and round-trips.

3. An unbiased estimator of the entropy rate#

• Why naive estimation fails: plug-in bias, wide alphabet, long memory.
• Stationary conditional sampling. Gibbs on the latent $x$ under rounding constraints; started from the generating $x$, which is an exact draw from $p(x \mid z)$, so no burn-in bias. Each sweep emits one exact sample of $z_{M+1}$.
• Unbiased entropy of the chain's marginal by Rhee–Glynn randomized telescoping with antithetic half-block corrections. Statement of unbiasedness; proof in Appendix A.
• Averaging over independent pasts gives a valid standard error; the only remaining approximation is finite $M$.
• Diagnostics and cost. (Table 1)

4. An analytic approximation#

• Szegő–Kolmogorov, its two failures (spectral nulls; only the quantized past is observed), and the 1/12 dither fix giving $s_*^2 = \exp\int \ln(S + 1/12) - 1/12$.
• $\bar H \approx G(s_*)$, the entropy of Gaussian ⊕ unit uniform.
• $s_*$ is computable from any signal's periodogram — no fit, no sampling.
• Agreement with §3 across the regime map. (Figure 1) Where it degrades and why.

5. Benchmark on synthetic sources#

• Codec families, all round-trip verified, all charged for side information: general-purpose byte compressors (± delta, ± shuffle); lossless audio (FLAC, WavPack, …); array/neuro pipelines; transform coders.
• Ratio-to-limit across the parameter space. (Figure 2) Which families win where; how far the best codec is from $\bar H$.
• 5.x LPC+ANS. Integer LPC with rANS over the empirical residual histogram. Absent from the ephys benchmarking literature; measures closest of the practical codecs. Method stated completely enough to reimplement.
• Advanced codecs approach the limit. A conditional-Gaussian arithmetic coder emits bytes that decode at 0.718 bits/sample where the estimator says 0.62–0.68 — an upper bound proven independently of §3. Impractical, included as evidence. (Table 2: the ladder.)

6. Extracellular voltage recordings#

Both signals are in scope: the raw wideband trace as acquired, and the bandpass-filtered trace prepared for spike sorting. Both are stored and shared in practice, and they sit in different regimes — measured $s_*$ is 3.8–6.8 raw against 0.91–1.10 filtered — so the codec ranking and the size of the shortfall differ between them. Report the two side by side throughout.

6.1 The limit transfers#

• Fit the model to three DANDI traces, raw and filtered, from their power spectrum alone.
• The surrogate has no spikes (kurtosis 3.0 vs 4.5–19.2) yet reproduces every prediction-based codec to within 0.05 bits/sample, on both variants. Note that the raw traces were never preprocessed — the fitted $h$ is a spectral factor, not a filter anyone applied. Compressibility is set by the noise spectrum, not the events.
• Domain of validity: noise $\gtrsim$ half a quantization step.

6.2 Where deployed codecs sit#

• Compute the limit for each recording and variant; measure what is actually used; report the shortfall. (Figure 3, two panels: raw and filtered.)
• The two variants give different answers, and the difference is worth stating plainly: on filtered traces LPC+ANS is ~21% below FLAC, while on raw traces the two are within 1–3%.
• Behaviour as the quantizer is coarsened — the near-lossless knob practitioners already use.

7. Discussion and limitations#

Stationarity and the noise condition are the binding assumptions — Gaussianity and filtering are not. Single channel. What the paper does not claim: no codec recommendation, no explanation of the shortfall.

8. Artifacts#

timeseries-entropy (estimator, theory, CLI); benchmark harness and codecs; interactive browser app; DANDI provenance.

---

Figures and tables#

#	Content
Fig 1	Analytic $G(s_*)$ vs Monte-Carlo $\bar H$ across the regime map, with error bars
Fig 2	Ratio-to-limit for each codec family across the synthetic parameter space
Fig 3	Real recordings: measured codec rates against the fitted-model limit
Tab 1	Estimator diagnostics: pilot decay, $M$-convergence, cost
Tab 2	The ladder at a reference point: raw → order-0 → FLAC → LPC+ANS → cond-Gaussian → $\bar H$

Appendices#

• A. Rhee–Glynn telescoping: unbiasedness proof, variance conditions.
• B. Gibbs conditionals for the box-truncated latent.
• C. Derivation of $G(s)$.
• D. Conditional-Gaussian codec spec.
• E. Spectral fitting for §6.
• F. Codec configurations and versions.