21abc12audio-spike-sorting: benchmark framework for audio event detection and classificationJeremy Magland 1# audio-spike-sorting
3A benchmark framework for the detection and classification of short audio
4events, in the spirit of spike sorting. You record hundreds of examples of
5each event type (tapping the table, tapping the floor, tapping with a
6spoon), cleanly separated so the examples are unambiguous. The framework
7then synthesizes recordings in which known events are placed at known
8times, possibly overlapping, hands each candidate algorithm a small
9labeled training set, and scores the algorithm's output against the exact
10ground truth. Candidate algorithms are written in MATLAB syntax and
11executed in the browser by [numbl](https://numbl.org); everything, from
12recording through scoring and visual exploration, runs on one page with
13nothing to install.
15A limitation of evaluating event-detection methods on real recordings is
16that the ground truth is itself the product of some annotation process,
17with its own errors and biases. Synthesis from recorded examples avoids
18this: the event times and labels are correct by construction, while the
19audio itself remains real. The trade-off is that a synthesized mixture is
20not a real performance; in particular, overlapping events here are sums of
21independently recorded events, which ignores any physical interaction
22between them.
24## How a benchmark run works
261. **Snippets.** Each recorded clip is segmented by an onset detector
27 (spectral flux, with an adjustable sensitivity), and the detected
28 events stay editable: click to audition, exclude mistakes. Every kept
29 onset is refined to the foot of its energy rise and a fixed window
30 (default 20 ms before, 180 ms after) is cut around it. This refined
31 onset is the project's single definition of event time, used both for
32 training snippets and for ground truth, so all algorithms are scored
33 against one consistent convention.
342. **Split.** For each run, every type's snippets are randomly divided
35 into a training set (default 30%) given to the algorithm, and a
36 placement pool used to build the dataset. An algorithm is therefore
37 always classifying events it has not literally seen.
383. **Synthesis.** Events from the placement pool are placed on a silent
e06c5e1Default to original event volumes; rename gain knob to level variation with an explanatory tooltipJeremy Magland 39 timeline at their recorded volumes. A level-variation knob can
40 optionally scale each placed event by a random gain drawn between
41 0 dB and a floor (say -12 dB), so that loudness carries no
42 information about identity; at the default of 0 dB no adjustment is
43 made. An overlap knob in [0, 1] sets the fraction of events
44 deliberately placed inside another event's span; the rest keep at
45 least one snippet length of separation. Run r uses seed + r, so the
46 whole benchmark is reproducible from the configuration.
21abc12audio-spike-sorting: benchmark framework for audio event detection and classificationJeremy Magland 474. **Execution.** Each algorithm is a `.m` file defining
48 `function [times, labels] = name(x, fs, training)`, where `training`
49 carries the per-type snippet matrices, the type names, and the onset
50 position within a snippet. numbl runs it in a web worker; the sources
51 are editable in place on the page.
525. **Scoring.** Detections are matched to ground truth greedily by time
53 difference within a tolerance (default 20 ms), ignoring labels. A
54 matched pair with the right label is correct, with the wrong label a
55 misclassification; unmatched truth events are misses and unmatched
56 detections false positives. The summary accuracy is
57 correct / (correct + misclassified + missed + false positives),
58 reported as mean and standard deviation over runs, alongside a
59 confusion matrix per algorithm with an extra column for misses and an
60 extra row for false positives.
16328b9Download runs as a .zip for offline development: WAVs, ground truth, training snippets, algorithm sources, and a numbl evaluation harness that reproduces the browser scoresJeremy Magland 62A completed benchmark can be downloaded as a .zip for offline algorithm
63development: every run's recording as a float32 WAV, its ground truth as
64a CSV, the exact training snippets it handed to the algorithms, the
65algorithm sources, and an evaluation harness (evaluate.m plus tools/)
66that the numbl CLI executes with `npx numbl run evaluate.m`, printing
67the same scores as the browser. The archive's README documents the file
68formats and the algorithm interface, so an algorithm developed against
69it drops straight back into the app.
21abc12audio-spike-sorting: benchmark framework for audio event detection and classificationJeremy Magland 71The explorer shows any synthesized recording with the ground truth lanes
72above the waveform and one algorithm's detections below it. A correct
73detection is a filled marker in its type's color, a misclassification is
74ringed in red, a false positive is a red cross, and a missed truth event
75is a hollow circle. The wheel zooms, dragging pans, and a click auditions
d6b5949Explorer playback: play the run or a shift-drag selected interval, with a tracking playheadJeremy Magland 76the audio at that time. The whole recording can also be played through,
77with a playhead tracking the sound: shift-drag selects an interval and
78the play button then plays just that selection, which is the quickest way
79to listen closely to a cluster of errors; a shift-click clears it.
21abc12audio-spike-sorting: benchmark framework for audio event detection and classificationJeremy Magland 80
81## Baseline algorithms
32dec4fAdd silence_gap_knn as a built-in; stop tracking the tmp/ offline workspaceJeremy Magland 83Four built-ins seed the comparison and can be edited or reset in place:
21abc12audio-spike-sorting: benchmark framework for audio event detection and classificationJeremy Magland 84
85- **template_matched_filter**: one unit-norm mean template per type,
86 cross-correlated against the recording via the FFT; the per-type scores
87 are normalized so one threshold treats loud and quiet types alike. The
88 classic spike-sorting approach; it handles nothing specially for
89 overlaps beyond the correlation itself.
90- **onset_knn**: detection on a smoothed energy envelope, classification
91 by distance-weighted k-nearest-neighbours over spectral shape features
92 (band-energy ratios, centroid, zero-crossing rate). Levels are used only
e06c5e1Default to original event volumes; rename gain knob to level variation with an explanatory tooltipJeremy Magland 93 as ratios, so level variation, when enabled, carries no information.
32dec4fAdd silence_gap_knn as a built-in; stop tracking the tmp/ offline workspaceJeremy Magland 94- **silence_gap_knn**: segmentation on runs of nonzero samples (the
95 synthesized background is digitally silent), then the same kNN
96 classifier as onset_knn. Developed offline against a downloaded
97 archive; the silence assumption is its stated limitation, and added
98 background noise or overlaps defeat the segmentation.
21abc12audio-spike-sorting: benchmark framework for audio event detection and classificationJeremy Magland 99- **naive_threshold**: an amplitude threshold and a classifier that looks
100 only at event duration. A deliberately weak floor; a real method that
101 fails to clear it is in trouble.
e06c5e1Default to original event volumes; rename gain knob to level variation with an explanatory tooltipJeremy Magland 103On the synthetic demo data (three types, 20% overlap, 12 dB of level
104variation), the baselines land around 0.6 to 0.8 accuracy, with the
105losses concentrated in the overlapping events. None of the built-ins attempts overlap resolution
21abc12audio-spike-sorting: benchmark framework for audio event detection and classificationJeremy Magland 106(subtraction or joint fitting), which is exactly the direction candidate
107algorithms are expected to explore.
109## Running it
111```bash
112npm install
113npm run dev # the app, at the printed URL
114npm test # node end-to-end test of the pipeline (no browser)
115npm run build # typecheck and bundle
116```
118"Load demo data" creates three synthetic event types so the whole
16328b9Download runs as a .zip for offline development: WAVs, ground truth, training snippets, algorithm sources, and a numbl evaluation harness that reproduces the browser scoresJeremy Magland 119pipeline can be tried without a microphone.
121`npm run test:offline` exercises the dataset download end to end: it
122builds the archive's files in node, runs the shipped evaluate.m through
123the numbl CLI, and requires the printed per-run accuracies to match the
124in-process scores exactly. Projects persist in the
21abc12audio-spike-sorting: benchmark framework for audio event detection and classificationJeremy Magland 125browser (clip audio in IndexedDB, everything else in localStorage) and
126round-trip through Export/Import as a single JSON file, so a recorded
127snippet library can be shared or committed.
129## Notes and limitations
131- The internal rate is 16 kHz mono; microphone input is downsampled on
132 capture with a box average, which is a crude anti-alias filter. Events
133 whose identity lives above 8 kHz will lose that information.
134- Synthesis adds no background noise; the noise floor is whatever the
135 recordings carry. A noise knob is a natural next step, as is an option
136 to place events over a recorded room-tone bed.
137- The matching is greedy rather than globally optimal; with a 20 ms
138 tolerance and events at realistic densities the difference is
139 negligible, but it is not the Hungarian algorithm.
140- Algorithm execution is single-threaded per run, so per-algorithm
141 timings are comparable but a benchmark with many runs takes their sum.