update paper
1 changed file+27−13
paper/paper.mdmodified+27−13View file
@@ -12,17 +12,7 @@
1212
1313 ## Introduction
1414
15-This is the introduction.
16-
17-## Methods
18-
19-### Compression Algorithms
20-
21-Describe the various compression methods.
22-
23-### Dataset Generation
24-
25-Describe the datasets we use for benchmarking.
15+[To be added]
2616
2717 ## Theory
2818
@@ -39,9 +29,33 @@ bits per sample. This means that the optimal compression ratio for such a datase
3929
4030 In practice, achieving this theoretical compression ratio requires sophisticated encoding techniques. Arithmetic encoding [ref] is one such method, but it is challenging to implement and can be computationally inefficient. A more modern and efficient alternative is Asymmetric Numeric Systems (ANS) [ref], which closely approaches the theoretical limit and is incorporated into state-of-the-art compressors such as ZStandard [ref]. However, these algorithms are primarily optimized for structured data types, such as text, rather than for numeric scientific data.
4131
42-In our benchmarks, we evaluate a simple implementation of ANS using a Python package we developed, called `simple_ans`. As anticipated, ANS demonstrates superior performance when compressing i.i.d. samples from a discrete distribution. However, its efficiency diminishes when handling more structured data, such as continuous signals (e.g., voltage traces in electrophysiology).
32+In our benchmarks, we evaluate a simple implementation of ANS using a Python package called `simple_ans`. As anticipated, ANS demonstrates superior performance when compressing i.i.d. samples from a discrete distribution. However, its efficiency diminishes when handling more structured data, such as continuous signals (e.g., voltage traces in electrophysiology).
33+
34+Applying delta encoding partially mitigates this limitation by leveraging the continuity properties of the data through differencing. This preprocessing step enhances ANS performance, though it still falls short of the compression achieved by methods like ZStandard. Additional preprocessing techniques, such as linear Markov predictive modeling, further improve ANS performance by exploiting temporal correlations in the data.
35+
36+The Markov prediction scheme employs a linear autoregressive model where each sample is predicted as a linear combination of $M$ previous samples. For a given integer time series $x[t]$, the prediction $\hat{x}[t]$ is computed as:
37+
38+$$
39+\hat{x}[t] = \text{round}\left(\sum_{i=1}^M c_i x[t-i] + b\right)
40+$$
41+
42+where the coefficients $c_i$ and bias term $b$ are determined through least squares regression on a subset of the data. The rounding operation ensures integer predictions. Rather than compressing the original signal directly, the algorithm compresses the integer residual error sequence $r[t] = x[t] - \hat{x}[t]$. This approach proves effective because the residuals typically have smaller magnitude compared with the original signal or with the deltas. The compression process stores three components: the floating-point model coefficients, a small set of initial integer values required to begin prediction, and the compressed integer residual sequence. During reconstruction, the original signal is recovered by computing predictions and adding back the residuals:
43+
44+$$
45+x[t] = \text{round}\left(\sum_{i=1}^M c_i x[t-i] + b\right) + r[t]
46+$$
47+
48+This predictive preprocessing step improves compression performance compared to applying ANS (or other algorithms) directly to either the raw signal or delta-encoded data.
49+
50+## Methods
4351
44-Applying delta encoding partially mitigates this limitation by leveraging the continuity properties of the data through differencing. This preprocessing step enhances ANS performance, though it still falls short of the compression achieved by methods like ZStandard. Additional preprocessing techniques, such as linear Markov predictive modeling (where the residual error after prediction is compressed instead of the original signal), further improve ANS performance. In these scenarios, the residual data is smaller and exhibits reduced correlation, enabling ANS to achieve better compression results relative to other methods.
52+### Compression Algorithms
53+
54+Describe the various compression methods.
55+
56+### Dataset Generation
57+
58+Describe the datasets we use for benchmarking.
4559
4660 ## Implementation
4761