Start the paper outline
Thesis: the entropy rate of this signal class is computable, and here is
where real codecs sit relative to it. The unbiased Monte-Carlo estimator is
the contribution; the analytic formula is a fast approximation validated
against it. Descriptive only — why codecs fall short is a separate paper.
Scope covers both the raw wideband and the bandpass-filtered ephys signal,
which land in different regimes and give different codec rankings.
1 changed file+176−0
paper/outline.mdadded+176−0View file
@@ -0,0 +1,176 @@
1+# Paper outline
2+
3+## Title
4+
5+The entropy rate of a noisy quantized signal and how far lossless codecs sit from it, with application to extracellular voltage recordings
6+
7+## Thesis
8+
9+The entropy rate of this signal class is computable. Here is where real codecs
10+sit relative to it.
11+
12+**Scope decisions.** The unbiased Monte-Carlo estimator is the contribution;
13+the analytic formula is a fast approximation validated against it. The paper is
14+descriptive — *why* codecs fall short (the gap decomposition) is a separate
15+paper. Target ~10–12 pages: estimator construction and the unbiasedness
16+statement in the body, proof in an appendix.
17+
18+**The estimator earns trust three ways:** it is provably unbiased; it agrees
19+with the theoretically justified analytic formula; and advanced codecs approach
20+it from above.
21+
22+---
23+
24+## 1. Introduction
25+
26+- Instruments emit quantized samples of a noisy continuous process. Lossless
27+ compression is a real cost, and the practice is empirical — try codecs, keep
28+ the winner. Nobody knows how much is left.
29+
30+### 1.1 Prior work
31+
32+*Organized around what each body of work supplies and what it leaves open.*
33+
34+- **Ephys compression benchmarks.** Buccino et al. benchmarked general-purpose
35+ and audio codecs on large-scale Neuropixels data, finding the audio codecs
36+ (WavPack, FLAC) outperform general-purpose compressors and that lossy modes
37+ buy substantial further ratio without measurably harming spike sorting. It is
38+ the reference point for this paper — **and it is codec-versus-codec.** With
39+ no entropy-rate estimate, "best" can only mean best of those tried, and the
40+ distance to what is achievable is unknown. Same for the surrounding
41+ literature on ephys and scientific-array compression (Blosc, Zarr/HDF5 filter
42+ pipelines).
43+- **Lossless audio coding.** FLAC, Shorten, WavPack, MPEG-4 ALS — the
44+ predict-then-code-the-residual architecture the ephys benchmarks inherit, and
45+ its Rice/Golomb residual coding.
46+- **Entropy-rate estimation.** Plug-in block entropy and its downward bias;
47+ Lempel–Ziv and compression-based estimators. The last are upper bounds of
48+ unknown tightness, so they cannot serve as the reference against which
49+ compressors are judged — using a compressor to estimate the limit that a
50+ compressor is being measured against begs the question.
51+- **Unbiased MCMC.** Rhee–Glynn randomized truncation; Jacob et al. — the
52+ machinery §3 builds on.
53+- **Quantization theory.** Bennett's dither model, Gray's quantization theory,
54+ high-resolution limits; the source of the 1/12 correction in §4.
55+
56+### 1.2 This paper
57+
58+- The missing quantity is the entropy rate $\bar H$. For this class it is not
59+ analytically available (rounding destroys Gaussianity) and not reliably
60+ estimable by plug-in methods (wide alphabet, long memory).
61+- We give an unbiased estimator of $\bar H$, an analytic approximation that
62+ agrees with it, and a benchmark of standard codecs against it — on synthetic
63+ sources with an exact limit, and on real extracellular recordings, both raw
64+ wideband and bandpass-filtered.
65+- Headline numbers to preview.
66+- Contributions.
67+
68+## 2. Setup
69+
70+- $z_t = \operatorname{round}(y_t)$, $y$ stationary Gaussian with PSD $S(f)$,
71+ unit quantization step. Amplitude in quantization steps; $S$ says everything.
72+- $y = h * x$ is the spectral-factorization form — $h$ is the minimum-phase
73+ factor of $S$, not a filter anyone applied.
74+- Estimand: $\bar H = \lim_{M\to\infty} H(z_{M+1} \mid z_1..z_M)$.
75+- Accounting: every reported size is the full encoded byte count, includes side
76+ information, and round-trips.
77+
78+## 3. An unbiased estimator of the entropy rate
79+
80+- **Why naive estimation fails**: plug-in bias, wide alphabet, long memory.
81+- **Stationary conditional sampling.** Gibbs on the latent $x$ under rounding
82+ constraints; started from the generating $x$, which is an exact draw from
83+ $p(x \mid z)$, so no burn-in bias. Each sweep emits one exact sample of
84+ $z_{M+1}$.
85+- **Unbiased entropy of the chain's marginal** by Rhee–Glynn randomized
86+ telescoping with antithetic half-block corrections. Statement of
87+ unbiasedness; proof in Appendix A.
88+- **Averaging over independent pasts** gives a valid standard error; the only
89+ remaining approximation is finite $M$.
90+- Diagnostics and cost. *(Table 1)*
91+
92+## 4. An analytic approximation
93+
94+- Szegő–Kolmogorov, its two failures (spectral nulls; only the quantized past
95+ is observed), and the 1/12 dither fix giving
96+ $s_*^2 = \exp\int \ln(S + 1/12) - 1/12$.
97+- $\bar H \approx G(s_*)$, the entropy of Gaussian ⊕ unit uniform.
98+- $s_*$ is computable from any signal's periodogram — no fit, no sampling.
99+- **Agreement with §3 across the regime map.** *(Figure 1)* Where it degrades
100+ and why.
101+
102+## 5. Benchmark on synthetic sources
103+
104+- Codec families, all round-trip verified, all charged for side information:
105+ general-purpose byte compressors (± delta, ± shuffle); lossless audio
106+ (FLAC, WavPack, …); array/neuro pipelines; transform coders.
107+- **Ratio-to-limit across the parameter space.** *(Figure 2)* Which families
108+ win where; how far the best codec is from $\bar H$.
109+- **5.x LPC+ANS.** Integer LPC with rANS over the empirical residual
110+ histogram. Absent from the ephys benchmarking literature; measures closest
111+ of the practical codecs. Method stated completely enough to reimplement.
112+- **Advanced codecs approach the limit.** A conditional-Gaussian arithmetic
113+ coder emits bytes that decode at 0.718 bits/sample where the estimator says
114+ 0.62–0.68 — an upper bound proven independently of §3. Impractical, included
115+ as evidence. *(Table 2: the ladder.)*
116+
117+## 6. Extracellular voltage recordings
118+
119+**Both signals are in scope**: the raw wideband trace as acquired, and the
120+bandpass-filtered trace prepared for spike sorting. Both are stored and shared
121+in practice, and they sit in different regimes — measured $s_*$ is 3.8–6.8 raw
122+against 0.91–1.10 filtered — so the codec ranking and the size of the shortfall
123+differ between them. Report the two side by side throughout.
124+
125+### 6.1 The limit transfers
126+
127+- Fit the model to three DANDI traces, raw and filtered, from their power
128+ spectrum alone.
129+- The surrogate has no spikes (kurtosis 3.0 vs 4.5–19.2) yet reproduces every
130+ prediction-based codec to within 0.05 bits/sample, on both variants. Note
131+ that the raw traces were never preprocessed — the fitted $h$ is a spectral
132+ factor, not a filter anyone applied. Compressibility is set by the noise
133+ spectrum, not the events.
134+- Domain of validity: noise $\gtrsim$ half a quantization step.
135+
136+### 6.2 Where deployed codecs sit
137+
138+- Compute the limit for each recording and variant; measure what is actually
139+ used; report the shortfall. *(Figure 3, two panels: raw and filtered.)*
140+- The two variants give different answers, and the difference is worth stating
141+ plainly: on filtered traces LPC+ANS is ~21% below FLAC, while on raw traces
142+ the two are within 1–3%.
143+- Behaviour as the quantizer is coarsened — the near-lossless knob
144+ practitioners already use.
145+
146+## 7. Discussion and limitations
147+
148+Stationarity and the noise condition are the binding assumptions — Gaussianity
149+and filtering are not. Single channel. What the paper does not claim: no codec
150+recommendation, no explanation of the shortfall.
151+
152+## 8. Artifacts
153+
154+`timeseries-entropy` (estimator, theory, CLI); benchmark harness and codecs;
155+interactive browser app; DANDI provenance.
156+
157+---
158+
159+## Figures and tables
160+
161+| # | Content |
162+|---|---|
163+| Fig 1 | Analytic $G(s_*)$ vs Monte-Carlo $\bar H$ across the regime map, with error bars |
164+| Fig 2 | Ratio-to-limit for each codec family across the synthetic parameter space |
165+| Fig 3 | Real recordings: measured codec rates against the fitted-model limit |
166+| Tab 1 | Estimator diagnostics: pilot decay, $M$-convergence, cost |
167+| Tab 2 | The ladder at a reference point: raw → order-0 → FLAC → LPC+ANS → cond-Gaussian → $\bar H$ |
168+
169+## Appendices
170+
171+- **A.** Rhee–Glynn telescoping: unbiasedness proof, variance conditions.
172+- **B.** Gibbs conditionals for the box-truncated latent.
173+- **C.** Derivation of $G(s)$.
174+- **D.** Conditional-Gaussian codec spec.
175+- **E.** Spectral fitting for §6.
176+- **F.** Codec configurations and versions.