1# audio-spike-sorting
3A benchmark framework for the detection and classification of short audio
4events, in the spirit of spike sorting. You record hundreds of examples of
5each event type (tapping the table, tapping the floor, tapping with a
6spoon), cleanly separated so the examples are unambiguous. The framework
7then synthesizes recordings in which known events are placed at known
8times, possibly overlapping, hands each candidate algorithm a small
9labeled training set, and scores the algorithm's output against the exact
10ground truth. Candidate algorithms are written in MATLAB syntax and
11executed in the browser by [numbl](https://numbl.org); everything, from
12recording through scoring and visual exploration, runs on one page with
13nothing to install.
15A limitation of evaluating event-detection methods on real recordings is
16that the ground truth is itself the product of some annotation process,
17with its own errors and biases. Synthesis from recorded examples avoids
18this: the event times and labels are correct by construction, while the
19audio itself remains real. The trade-off is that a synthesized mixture is
20not a real performance; in particular, overlapping events here are sums of
21independently recorded events, which ignores any physical interaction
22between them.
24## How a benchmark run works
261. **Snippets.** Each recorded clip is segmented by an onset detector
27 (spectral flux, with an adjustable sensitivity), and the detected
28 events stay editable: click to audition, exclude mistakes. Every kept
29 onset is refined to the foot of its energy rise and a fixed window
30 (default 20 ms before, 180 ms after) is cut around it. This refined
31 onset is the project's single definition of event time, used both for
32 training snippets and for ground truth, so all algorithms are scored
33 against one consistent convention.
342. **Split.** For each run, every type's snippets are randomly divided
35 into a training set (default 30%) given to the algorithm, and a
36 placement pool used to build the dataset. An algorithm is therefore
37 always classifying events it has not literally seen.
383. **Synthesis.** Events from the placement pool are placed on a silent
39 timeline at their recorded volumes. A level-variation knob can
40 optionally scale each placed event by a random gain drawn between
41 0 dB and a floor (say -12 dB), so that loudness carries no
42 information about identity; at the default of 0 dB no adjustment is
43 made. An overlap knob in [0, 1] sets the fraction of events
44 deliberately placed inside another event's span; the rest keep at
45 least one snippet length of separation. Run r uses seed + r, so the
46 whole benchmark is reproducible from the configuration.
474. **Execution.** Each algorithm is a `.m` file defining
48 `function [times, labels] = name(x, fs, training)`, where `training`
49 carries the per-type snippet matrices, the type names, and the onset
50 position within a snippet. numbl runs it in a web worker; the sources
51 are editable in place on the page.
525. **Scoring.** Detections are matched to ground truth greedily by time
53 difference within a tolerance (default 20 ms), ignoring labels. A
54 matched pair with the right label is correct, with the wrong label a
55 misclassification; unmatched truth events are misses and unmatched
56 detections false positives. The summary accuracy is
57 correct / (correct + misclassified + missed + false positives),
58 reported as mean and standard deviation over runs, alongside a
59 confusion matrix per algorithm with an extra column for misses and an
60 extra row for false positives.
62A completed benchmark can be downloaded as a .zip for offline algorithm
63development: every run's recording as a float32 WAV, its ground truth as
64a CSV, the exact training snippets it handed to the algorithms, the
65algorithm sources, and an evaluation harness (evaluate.m plus tools/)
66that the numbl CLI executes with `npx numbl run evaluate.m`, printing
67the same scores as the browser. The archive's README documents the file
68formats and the algorithm interface, so an algorithm developed against
69it drops straight back into the app.
71The explorer shows any synthesized recording with the ground truth lanes
72above the waveform and one algorithm's detections below it. A correct
73detection is a filled marker in its type's color, a misclassification is
74ringed in red, a false positive is a red cross, and a missed truth event
75is a hollow circle. The wheel zooms, dragging pans, and a click auditions
76the audio at that time. The whole recording can also be played through,
77with a playhead tracking the sound: shift-drag selects an interval and
78the play button then plays just that selection, which is the quickest way
79to listen closely to a cluster of errors; a shift-click clears it.
81## Baseline algorithms
83Three built-ins seed the comparison and can be edited or reset in place:
85- **template_matched_filter**: one unit-norm mean template per type,
86 cross-correlated against the recording via the FFT; the per-type scores
87 are normalized so one threshold treats loud and quiet types alike. The
88 classic spike-sorting approach; it handles nothing specially for
89 overlaps beyond the correlation itself.
90- **onset_knn**: detection on a smoothed energy envelope, classification
91 by distance-weighted k-nearest-neighbours over spectral shape features
92 (band-energy ratios, centroid, zero-crossing rate). Levels are used only
93 as ratios, so level variation, when enabled, carries no information.
94- **naive_threshold**: an amplitude threshold and a classifier that looks
95 only at event duration. A deliberately weak floor; a real method that
96 fails to clear it is in trouble.
98On the synthetic demo data (three types, 20% overlap, 12 dB of level
99variation), the baselines land around 0.6 to 0.8 accuracy, with the
100losses concentrated in the overlapping events. None of the built-ins attempts overlap resolution
101(subtraction or joint fitting), which is exactly the direction candidate
102algorithms are expected to explore.
104## Running it
106```bash
107npm install
108npm run dev # the app, at the printed URL
109npm test # node end-to-end test of the pipeline (no browser)
110npm run build # typecheck and bundle
111```
113"Load demo data" creates three synthetic event types so the whole
114pipeline can be tried without a microphone.
116`npm run test:offline` exercises the dataset download end to end: it
117builds the archive's files in node, runs the shipped evaluate.m through
118the numbl CLI, and requires the printed per-run accuracies to match the
119in-process scores exactly. Projects persist in the
120browser (clip audio in IndexedDB, everything else in localStorage) and
121round-trip through Export/Import as a single JSON file, so a recorded
122snippet library can be shared or committed.
124## Notes and limitations
126- The internal rate is 16 kHz mono; microphone input is downsampled on
127 capture with a box average, which is a crude anti-alias filter. Events
128 whose identity lives above 8 kHz will lose that information.
129- Synthesis adds no background noise; the noise floor is whatever the
130 recordings carry. A noise knob is a natural next step, as is an option
131 to place events over a recorded room-tone bed.
132- The matching is greedy rather than globally optimal; with a 20 ms
133 tolerance and events at realistic densities the difference is
134 negligible, but it is not the Hungarian algorithm.
135- Algorithm execution is single-threaded per run, so per-algorithm
136 timings are comparable but a benchmark with many runs takes their sum.