paper refs
2 changed files+68−16
paper/paper.mdmodified+3−3View file
@@ -29,13 +29,13 @@ To help researchers navigate these questions, we introduce Benchcompress, a benc
2929
3030 ## Traditional Compression Methods and Scientific Data
3131
32-Traditional compression algorithms like LZW (Lempel-Ziv-Welch), LZ77, BWT (Burrows-Wheeler Transform), and Huffman coding were originally designed for text data where patterns manifest as repeated sequences of characters or words. While these methods can compress numeric data from scientific instruments, they may not be optimal for this use case. Text compression excels at identifying exact matches of recurring patterns, whereas scientific measurements often exhibit more complex relationships between values.
32+Traditional compression algorithms like LZW (Lempel-Ziv-Welch) [@welch1984technique], LZ77 [@ziv1977universal], BWT (Burrows-Wheeler Transform) [@burrows1994block], and Huffman coding [@huffman1952method] were originally designed for text data where patterns manifest as repeated sequences of characters or words. While these methods can compress numeric data from scientific instruments, they may not be optimal for this use case. Text compression excels at identifying exact matches of recurring patterns, whereas scientific measurements often exhibit more complex relationships between values.
3333
3434 For example, in text compression, finding repeated instances of common words or phrases leads to efficient encoding. In contrast, scientific data arrays often contain continuous variations where measurement noise and uncertainties mean exact repetition is rare. Even when numeric values are similar, their byte-level representations may share some common bits while differing in others. This characteristic of scientific data suggests that methods explicitly accounting for numerical relationships between values may achieve better compression ratios than general-purpose algorithms.
3535
3636 ## Independent identically distributed samples
3737
38-Moving from text to numeric data, we begin with the fundamental case of independently and identically distributed (i.i.d.) discrete data. For such data, where each sample is drawn from a discrete probability distribution (e.g., Bernoulli sampling or quantized Gaussian noise), the theoretical compressed size in bits per sample is determined by the Shannon entropy formula:
38+Moving from text to numeric data, we begin with the fundamental case of independently and identically distributed (i.i.d.) discrete data. For such data, where each sample is drawn from a discrete probability distribution (e.g., Bernoulli sampling or quantized Gaussian noise), the theoretical compressed size in bits per sample is determined by Shannon's entropy formula [@shannon1948mathematical]:
3939
4040 $$
4141 H(X) = -\sum_{i} p(x_i) \log_2 p(x_i).
@@ -51,7 +51,7 @@ bits per sample. This means that the optimal compression ratio for such a datase
5151
5252 ## Asymmetric numeral systems
5353
54-In practice, achieving this theoretical compression ratio requires sophisticated encoding techniques. Arithmetic coding [@witten1987arithmetic] is one such method, but it is challenging to implement and can be computationally inefficient. A more modern and efficient alternative is asymmetric numeral systems (ANS) [@duda2013asymmetric], which closely approaches the theoretical limit and is incorporated into state-of-the-art compressors such as ZStandard [@zstd]. However, these large, general-use packages are primarily optimized for structured data types, such as text, rather than for numeric scientific data. In our benchmarks, we evaluate ANS using a simple, no-frills, implementation using a Python package we developed for this purpose called `simple_ans`. As anticipated, we show that ANS demonstrates superior performance when compressing i.i.d. samples from a discrete distribution.
54+In practice, achieving this theoretical compression ratio requires sophisticated encoding techniques. Arithmetic coding [@witten1987arithmetic] is one such method, but it is challenging to implement and can be computationally inefficient. A more modern and efficient alternative is asymmetric numeral systems (ANS) [@duda2015use], which closely approaches the theoretical limit and is incorporated into state-of-the-art compressors such as ZStandard [@collet2018zstandard]. However, these large, general-use packages are primarily optimized for structured data types, such as text, rather than for numeric scientific data. In our benchmarks, we evaluate ANS using a simple, no-frills, implementation using a Python package we developed for this purpose called `simple_ans`. As anticipated, we show that ANS demonstrates superior performance when compressing i.i.d. samples from a discrete distribution.
5555
5656 ## Delta encoding
5757
paper/references.bibmodified+65−13View file
@@ -6,19 +6,21 @@
66 number={6},
77 pages={520--540},
88 year={1987},
9- publisher={ACM}
9+ publisher={ACM New York, NY, USA}
1010 }
1111
12-@article{duda2013asymmetric,
13- title={Asymmetric numeral systems: entropy coding combining speed of Huffman coding with compression rate of arithmetic coding},
14- author={Duda, Jarek},
15- journal={arXiv preprint arXiv:1311.2540},
16- year={2013}
12+@inproceedings{duda2015use,
13+ title={The use of asymmetric numeral systems as an accurate replacement for Huffman coding},
14+ author={Duda, Jarek and Tahboub, Khalid and Gadgil, Neeraj J and Delp, Edward J},
15+ booktitle={2015 Picture Coding Symposium (PCS)},
16+ pages={65--69},
17+ year={2015},
18+ organization={IEEE}
1719 }
1820
1921 @article{buccino2023compression,
2022 title={Compression strategies for large-scale electrophysiology data},
21- author={Buccino, Alessio P and Winter, Oliver and Bryant, Dylan and Feng, Dong and Svoboda, Karel and Siegle, Josh H},
23+ author={Buccino, Alessio P and Winter, Olivier and Bryant, David and Feng, David and Svoboda, Karel and Siegle, Joshua H},
2224 journal={Journal of Neural Engineering},
2325 volume={20},
2426 number={5},
@@ -27,10 +29,60 @@
2729 publisher={IOP Publishing}
2830 }
2931
30-@misc{zstd,
31- title={Zstandard - Fast real-time compression algorithm},
32- author={{Meta Platforms, Inc.}},
33- year={2023},
34- howpublished={\url{https://github.com/facebook/zstd}},
35- note={Accessed: 2024-01-30}
32+@techreport{collet2018zstandard,
33+ title={Zstandard Compression and the application/zstd Media Type},
34+ author={Collet, Yann and Kucherawy, Murray},
35+ year={2018}
36+}
37+
38+@article{welch1984technique,
39+ title={A technique for high-performance data compression},
40+ author={Welch, Terry A.},
41+ journal={Computer},
42+ volume={17},
43+ number={06},
44+ pages={8--19},
45+ year={1984},
46+ publisher={IEEE Computer Society}
47+}
48+
49+@article{ziv1977universal,
50+ title={A universal algorithm for sequential data compression},
51+ author={Ziv, Jacob and Lempel, Abraham},
52+ journal={IEEE Transactions on information theory},
53+ volume={23},
54+ number={3},
55+ pages={337--343},
56+ year={1977},
57+ publisher={IEEE}
58+}
59+
60+@article{burrows1994block,
61+ title={A block-sorting lossless data compression algorithm},
62+ author={Burrows, Michael},
63+ journal={SRS Research Report},
64+ volume={124},
65+ year={1994}
66+}
67+
68+@article{shannon1948mathematical,
69+ title={A mathematical theory of communication},
70+ author={Shannon, Claude Elwood},
71+ journal={The Bell system technical journal},
72+ volume={27},
73+ number={3},
74+ pages={379--423},
75+ year={1948},
76+ publisher={Nokia Bell Labs}
77+}
78+
79+@article{huffman1952method,
80+ title={A method for the construction of minimum-redundancy codes},
81+ author={Huffman, David A},
82+ journal={Proceedings of the IRE},
83+ volume={40},
84+ number={9},
85+ pages={1098--1101},
86+ year={1952},
87+ publisher={IEEE}
3688 }