1# audio-spike-sorting
3A benchmark framework for the detection and classification of short audio
4events, in the spirit of spike sorting. You record hundreds of examples of
5each event type (tapping the table, tapping the floor, tapping with a
6spoon), cleanly separated so the examples are unambiguous. The framework
7then synthesizes recordings in which known events are placed at known
8times, possibly overlapping, hands each candidate algorithm a small
9labeled training set, and scores the algorithm's output against the exact
10ground truth. Candidate algorithms are written in MATLAB syntax and
11executed in the browser by [numbl](https://numbl.org); everything, from
12recording through scoring and visual exploration, runs on one page with
13nothing to install.
15A limitation of evaluating event-detection methods on real recordings is
16that the ground truth is itself the product of some annotation process,
17with its own errors and biases. Synthesis from recorded examples avoids
18this: the event times and labels are correct by construction, while the
19audio itself remains real. The trade-off is that a synthesized mixture is
20not a real performance; in particular, overlapping events here are sums of
21independently recorded events, which ignores any physical interaction
22between them.
24## How a benchmark run works
261. **Snippets.** Each recorded clip is segmented by an onset detector
27 (spectral flux, with an adjustable sensitivity), and the detected
28 events stay editable: click to audition, exclude mistakes. Every kept
29 onset is refined to the foot of its energy rise and a fixed window
30 (default 20 ms before, 180 ms after) is cut around it. This refined
31 onset is the project's single definition of event time, used both for
32 training snippets and for ground truth, so all algorithms are scored
33 against one consistent convention.
342. **Split.** For each run, every type's snippets are randomly divided
35 into a training set (default 30%) given to the algorithm, and a
36 placement pool used to build the dataset. An algorithm is therefore
37 always classifying events it has not literally seen.
383. **Synthesis.** Events from the placement pool are placed on a silent
39 timeline at their recorded volumes. A level-variation knob can
40 optionally scale each placed event by a random gain drawn between
41 0 dB and a floor (say -12 dB), so that loudness carries no
42 information about identity; at the default of 0 dB no adjustment is
43 made. An overlap knob in [0, 1] sets the fraction of events
44 deliberately placed inside another event's span; the rest keep at
45 least one snippet length of separation. A coincidence knob, also in
46 [0, 1], then sets what fraction of those overlapping events land on
47 their host's onset instead of at a random offset within its span,
48 within an adjustable window (default 5 ms either side). This is the
49 hardest case the synthesis can produce: two onsets no detector can
50 separate in time, and a mixture the classifier must take apart to
51 name both. It is off by default, so datasets from earlier
52 configurations reproduce unchanged. Run r uses seed + r, so the
53 whole benchmark is reproducible from the configuration.
544. **Execution.** Each algorithm is a `.m` file defining
55 `function [times, labels] = name(x, fs, training)`, where `training`
56 carries the per-type snippet matrices, the type names, and the onset
57 position within a snippet. numbl runs it in a web worker; the sources
58 are editable in place on the page.
595. **Scoring.** Detections are matched to ground truth greedily by time
60 difference within a tolerance (default 20 ms), ignoring labels. A
61 matched pair with the right label is correct, with the wrong label a
62 misclassification; unmatched truth events are misses and unmatched
63 detections false positives. The summary accuracy is
64 correct / (correct + misclassified + missed + false positives),
65 reported as mean and standard deviation over runs, alongside a
66 confusion matrix per algorithm with an extra column for misses and an
67 extra row for false positives.
69A completed benchmark can be downloaded as a .zip for offline algorithm
70development: every run's recording as a float32 WAV, its ground truth as
71a CSV, the exact training snippets it handed to the algorithms, the
72algorithm sources, and an evaluation harness (evaluate.m plus tools/)
73that the numbl CLI executes with `npx numbl run evaluate.m`, printing
74the same scores as the browser. The archive's README documents the file
75formats and the algorithm interface, so an algorithm developed against
76it drops straight back into the app.
78The explorer shows any synthesized recording with the ground truth lanes
79above the waveform and one algorithm's detections below it. A correct
80detection is a filled marker in its type's color, a misclassification is
81ringed in red, a false positive is a red cross, and a missed truth event
82is a hollow circle. The wheel zooms, dragging pans, and a click auditions
83the audio at that time. The whole recording can also be played through,
84with a playhead tracking the sound: shift-drag selects an interval and
85the play button then plays just that selection, which is the quickest way
86to listen closely to a cluster of errors; a shift-click clears it.
88## Baseline algorithms
90Four built-ins seed the comparison and can be edited or reset in place:
92- **template_matched_filter**: one unit-norm mean template per type,
93 cross-correlated against the recording via the FFT; the per-type scores
94 are normalized so one threshold treats loud and quiet types alike. The
95 classic spike-sorting approach; it handles nothing specially for
96 overlaps beyond the correlation itself.
97- **onset_knn**: detection on a smoothed energy envelope, classification
98 by distance-weighted k-nearest-neighbours over spectral shape features
99 (band-energy ratios, centroid, zero-crossing rate). Levels are used only
100 as ratios, so level variation, when enabled, carries no information.
101- **silence_gap_knn**: segmentation on runs of nonzero samples (the
102 synthesized background is digitally silent), then the same kNN
103 classifier as onset_knn. Developed offline against a downloaded
104 archive; the silence assumption is its stated limitation, and added
105 background noise or overlaps defeat the segmentation.
106- **naive_threshold**: an amplitude threshold and a classifier that looks
107 only at event duration. A deliberately weak floor; a real method that
108 fails to clear it is in trouble.
110On the synthetic demo data (three types, 20% overlap, 12 dB of level
111variation), the baselines land around 0.6 to 0.8 accuracy, with the
112losses concentrated in the overlapping events. Turning the coincidence
113knob up costs another few points: at 40% overlap on the same data,
114onset_knn falls from about 0.57 to 0.50 when every overlapping event is
115made coincident, since two onsets inside 5 ms present as one detection
116and the second event is scored as a miss. None of the built-ins attempts
117overlap resolution (subtraction or joint fitting), which is exactly the
118direction candidate algorithms are expected to explore, and coincident
119events are the case where that resolution has to be joint rather than
120sequential.
122## Running it
124```bash
125npm install
126npm run dev # the app, at the printed URL
127npm test # node end-to-end test of the pipeline (no browser)
128npm run build # typecheck and bundle
129```
131"Load demo data" creates three synthetic event types so the whole
132pipeline can be tried without a microphone.
134`npm run test:offline` exercises the dataset download end to end: it
135builds the archive's files in node, runs the shipped evaluate.m through
136the numbl CLI, and requires the printed per-run accuracies to match the
137in-process scores exactly. Projects persist in the
138browser (clip audio in IndexedDB, everything else in localStorage) and
139round-trip through Export/Import as a single JSON file, so a recorded
140snippet library can be shared or committed.
142## Notes and limitations
144- The internal rate is 16 kHz mono; microphone input is downsampled on
145 capture with a box average, which is a crude anti-alias filter. Events
146 whose identity lives above 8 kHz will lose that information.
147- Synthesis adds no background noise; the noise floor is whatever the
148 recordings carry. A noise knob is a natural next step, as is an option
149 to place events over a recorded room-tone bed.
150- The matching is greedy rather than globally optimal; with a 20 ms
151 tolerance and events at realistic densities the difference is
152 negligible, but it is not the Hungarian algorithm. Note that
153 coincident events are the case where greedy matching is least safe:
154 two truth events within the match tolerance of each other compete for
155 the same detections, and the closest-pair-first rule can assign a
156 detection to the wrong one of the pair. The effect is on which of two
157 simultaneous events is called correct, not on the totals.
158- A coincident event is placed on top of a host that may itself be
159 coincident, so at a high coincidence fraction the synthesis produces
160 occasional pile-ups of three or more events rather than only pairs.
161- Algorithm execution is single-threaded per run, so per-algorithm
162 timings are comparable but a benchmark with many runs takes their sum.