1# Paper outline
3## Title
5The entropy rate of a noisy quantized signal and how far lossless codecs sit from it, with application to extracellular voltage recordings
7## Thesis
9The entropy rate of this signal class is computable. Here is where real codecs
10sit relative to it.
12**Scope decisions.** The unbiased Monte-Carlo estimator is the contribution;
13the analytic formula is a fast approximation validated against it. The paper is
14descriptive — *why* codecs fall short (the gap decomposition) is a separate
15paper. Target ~10–12 pages: estimator construction and the unbiasedness
16statement in the body, proof in an appendix.
18**The estimator earns trust three ways:** it is provably unbiased; it agrees
19with the theoretically justified analytic formula; and advanced codecs approach
20it from above.
22---
24## 1. Introduction
26- Instruments emit quantized samples of a noisy continuous process. Lossless
27 compression is a real cost, and the practice is empirical — try codecs, keep
28 the winner. Nobody knows how much is left.
30### 1.1 Prior work
32*Organized around what each body of work supplies and what it leaves open.*
34- **Ephys compression benchmarks.** Buccino et al. benchmarked general-purpose
35 and audio codecs on large-scale Neuropixels data, finding the audio codecs
36 (WavPack, FLAC) outperform general-purpose compressors and that lossy modes
37 buy substantial further ratio without measurably harming spike sorting. It is
38 the reference point for this paper — **and it is codec-versus-codec.** With
39 no entropy-rate estimate, "best" can only mean best of those tried, and the
40 distance to what is achievable is unknown. Same for the surrounding
41 literature on ephys and scientific-array compression (Blosc, Zarr/HDF5 filter
42 pipelines).
43- **Lossless audio coding.** FLAC, Shorten, WavPack, MPEG-4 ALS — the
44 predict-then-code-the-residual architecture the ephys benchmarks inherit, and
45 its Rice/Golomb residual coding.
46- **Entropy-rate estimation.** Plug-in block entropy and its downward bias;
47 Lempel–Ziv and compression-based estimators. The last are upper bounds of
48 unknown tightness, so they cannot serve as the reference against which
49 compressors are judged — using a compressor to estimate the limit that a
50 compressor is being measured against begs the question.
51- **Unbiased MCMC.** Rhee–Glynn randomized truncation; Jacob et al. — the
52 machinery §3 builds on.
53- **Quantization theory.** Bennett's dither model, Gray's quantization theory,
54 high-resolution limits; the source of the 1/12 correction in §4.
56### 1.2 This paper
58- The missing quantity is the entropy rate $\bar H$. For this class it is not
59 analytically available (rounding destroys Gaussianity) and not reliably
60 estimable by plug-in methods (wide alphabet, long memory).
61- We give an unbiased estimator of $\bar H$, an analytic approximation that
62 agrees with it, and a benchmark of standard codecs against it — on synthetic
63 sources with an exact limit, and on real extracellular recordings, both raw
64 wideband and bandpass-filtered.
65- Headline numbers to preview.
66- Contributions.
68## 2. Setup
70- $z_t = \operatorname{round}(y_t)$, $y$ stationary Gaussian with PSD $S(f)$,
71 unit quantization step. Amplitude in quantization steps; $S$ says everything.
72- $y = h * x$ is the spectral-factorization form — $h$ is the minimum-phase
73 factor of $S$, not a filter anyone applied.
74- Estimand: $\bar H = \lim_{M\to\infty} H(z_{M+1} \mid z_1..z_M)$.
75- Accounting: every reported size is the full encoded byte count, includes side
76 information, and round-trips.
78## 3. An unbiased estimator of the entropy rate
80- **Why naive estimation fails**: plug-in bias, wide alphabet, long memory.
81- **Stationary conditional sampling.** Gibbs on the latent $x$ under rounding
82 constraints; started from the generating $x$, which is an exact draw from
83 $p(x \mid z)$, so no burn-in bias. Each sweep emits one exact sample of
84 $z_{M+1}$.
85- **Unbiased entropy of the chain's marginal** by Rhee–Glynn randomized
86 telescoping with antithetic half-block corrections. Statement of
87 unbiasedness; proof in Appendix A.
88- **Averaging over independent pasts** gives a valid standard error; the only
89 remaining approximation is finite $M$.
90- Diagnostics and cost. *(Table 1)*
92## 4. An analytic approximation
94- Szegő–Kolmogorov, its two failures (spectral nulls; only the quantized past
95 is observed), and the 1/12 dither fix giving
96 $s_*^2 = \exp\int \ln(S + 1/12) - 1/12$.
97- $\bar H \approx G(s_*)$, the entropy of Gaussian ⊕ unit uniform.
98- $s_*$ is computable from any signal's periodogram — no fit, no sampling.
99- **Agreement with §3 across the regime map.** *(Figure 1)* Where it degrades
100 and why.
102## 5. Benchmark on synthetic sources
104- Codec families, all round-trip verified, all charged for side information:
105 general-purpose byte compressors (± delta, ± shuffle); lossless audio
106 (FLAC, WavPack, …); array/neuro pipelines; transform coders.
107- **Ratio-to-limit across the parameter space.** *(Figure 2)* Which families
108 win where; how far the best codec is from $\bar H$.
109- **5.x LPC+ANS.** Integer LPC with rANS over the empirical residual
110 histogram. Absent from the ephys benchmarking literature; measures closest
111 of the practical codecs. Method stated completely enough to reimplement.
112- **Advanced codecs approach the limit.** A conditional-Gaussian arithmetic
113 coder emits bytes that decode at 0.718 bits/sample where the estimator says
114 0.62–0.68 — an upper bound proven independently of §3. Impractical, included
115 as evidence. *(Table 2: the ladder.)*
117## 6. Extracellular voltage recordings
119**Both signals are in scope**: the raw wideband trace as acquired, and the
120bandpass-filtered trace prepared for spike sorting. Both are stored and shared
121in practice, and they sit in different regimes — measured $s_*$ is 3.8–6.8 raw
122against 0.91–1.10 filtered — so the codec ranking and the size of the shortfall
123differ between them. Report the two side by side throughout.
125### 6.1 The limit transfers
127- Fit the model to three DANDI traces, raw and filtered, from their power
128 spectrum alone.
129- The surrogate has no spikes (kurtosis 3.0 vs 4.5–19.2) yet reproduces every
130 prediction-based codec to within 0.05 bits/sample, on both variants. Note
131 that the raw traces were never preprocessed — the fitted $h$ is a spectral
132 factor, not a filter anyone applied. Compressibility is set by the noise
133 spectrum, not the events.
134- Domain of validity: noise $\gtrsim$ half a quantization step.
136### 6.2 Where deployed codecs sit
138- Compute the limit for each recording and variant; measure what is actually
139 used; report the shortfall. *(Figure 3, two panels: raw and filtered.)*
140- The two variants give different answers, and the difference is worth stating
141 plainly: on filtered traces LPC+ANS is ~21% below FLAC, while on raw traces
142 the two are within 1–3%.
143- Behaviour as the quantizer is coarsened — the near-lossless knob
144 practitioners already use.
146## 7. Discussion and limitations
148Stationarity and the noise condition are the binding assumptions — Gaussianity
149and filtering are not. Single channel. What the paper does not claim: no codec
150recommendation, no explanation of the shortfall.
152## 8. Artifacts
154`timeseries-entropy` (estimator, theory, CLI); benchmark harness and codecs;
155interactive browser app; DANDI provenance.
157---
159## Figures and tables
161| # | Content |
162|---|---|
163| Fig 1 | Analytic $G(s_*)$ vs Monte-Carlo $\bar H$ across the regime map, with error bars |
164| Fig 2 | Ratio-to-limit for each codec family across the synthetic parameter space |
165| Fig 3 | Real recordings: measured codec rates against the fitted-model limit |
166| Tab 1 | Estimator diagnostics: pilot decay, $M$-convergence, cost |
167| Tab 2 | The ladder at a reference point: raw → order-0 → FLAC → LPC+ANS → cond-Gaussian → $\bar H$ |
169## Appendices
171- **A.** Rhee–Glynn telescoping: unbiasedness proof, variance conditions.
172- **B.** Gibbs conditionals for the box-truncated latent.
173- **C.** Derivation of $G(s)$.
174- **D.** Conditional-Gaussian codec spec.
175- **E.** Spectral fitting for §6.
176- **F.** Codec configurations and versions.