21abc12audio-spike-sorting: benchmark framework for audio event detection and classificationJeremy Magland 1# audio-spike-sorting
3A benchmark framework for the detection and classification of short audio
4events, in the spirit of spike sorting. You record hundreds of examples of
5each event type (tapping the table, tapping the floor, tapping with a
6spoon), cleanly separated so the examples are unambiguous. The framework
7then synthesizes recordings in which known events are placed at known
8times, possibly overlapping, hands each candidate algorithm a small
9labeled training set, and scores the algorithm's output against the exact
10ground truth. Candidate algorithms are written in MATLAB syntax and
11executed in the browser by [numbl](https://numbl.org); everything, from
12recording through scoring and visual exploration, runs on one page with
13nothing to install.
15A limitation of evaluating event-detection methods on real recordings is
16that the ground truth is itself the product of some annotation process,
17with its own errors and biases. Synthesis from recorded examples avoids
18this: the event times and labels are correct by construction, while the
19audio itself remains real. The trade-off is that a synthesized mixture is
20not a real performance; in particular, overlapping events here are sums of
21independently recorded events, which ignores any physical interaction
22between them.
24## How a benchmark run works
261. **Snippets.** Each recorded clip is segmented by an onset detector
27 (spectral flux, with an adjustable sensitivity), and the detected
28 events stay editable: click to audition, exclude mistakes. Every kept
29 onset is refined to the foot of its energy rise and a fixed window
30 (default 20 ms before, 180 ms after) is cut around it. This refined
31 onset is the project's single definition of event time, used both for
32 training snippets and for ground truth, so all algorithms are scored
33 against one consistent convention.
342. **Split.** For each run, every type's snippets are randomly divided
35 into a training set (default 30%) given to the algorithm, and a
36 placement pool used to build the dataset. An algorithm is therefore
37 always classifying events it has not literally seen.
383. **Synthesis.** Events from the placement pool are placed on a silent
e06c5e1Default to original event volumes; rename gain knob to level variation with an explanatory tooltipJeremy Magland 39 timeline at their recorded volumes. A level-variation knob can
40 optionally scale each placed event by a random gain drawn between
41 0 dB and a floor (say -12 dB), so that loudness carries no
42 information about identity; at the default of 0 dB no adjustment is
43 made. An overlap knob in [0, 1] sets the fraction of events
44 deliberately placed inside another event's span; the rest keep at
45 least one snippet length of separation. Run r uses seed + r, so the
46 whole benchmark is reproducible from the configuration.
21abc12audio-spike-sorting: benchmark framework for audio event detection and classificationJeremy Magland 474. **Execution.** Each algorithm is a `.m` file defining
48 `function [times, labels] = name(x, fs, training)`, where `training`
49 carries the per-type snippet matrices, the type names, and the onset
50 position within a snippet. numbl runs it in a web worker; the sources
51 are editable in place on the page.
525. **Scoring.** Detections are matched to ground truth greedily by time
53 difference within a tolerance (default 20 ms), ignoring labels. A
54 matched pair with the right label is correct, with the wrong label a
55 misclassification; unmatched truth events are misses and unmatched
56 detections false positives. The summary accuracy is
57 correct / (correct + misclassified + missed + false positives),
58 reported as mean and standard deviation over runs, alongside a
59 confusion matrix per algorithm with an extra column for misses and an
60 extra row for false positives.
62The explorer shows any synthesized recording with the ground truth lanes
63above the waveform and one algorithm's detections below it. A correct
64detection is a filled marker in its type's color, a misclassification is
65ringed in red, a false positive is a red cross, and a missed truth event
66is a hollow circle. The wheel zooms, dragging pans, and a click auditions
d6b5949Explorer playback: play the run or a shift-drag selected interval, with a tracking playheadJeremy Magland 67the audio at that time. The whole recording can also be played through,
68with a playhead tracking the sound: shift-drag selects an interval and
69the play button then plays just that selection, which is the quickest way
70to listen closely to a cluster of errors; a shift-click clears it.
21abc12audio-spike-sorting: benchmark framework for audio event detection and classificationJeremy Magland 71
72## Baseline algorithms
74Three built-ins seed the comparison and can be edited or reset in place:
76- **template_matched_filter**: one unit-norm mean template per type,
77 cross-correlated against the recording via the FFT; the per-type scores
78 are normalized so one threshold treats loud and quiet types alike. The
79 classic spike-sorting approach; it handles nothing specially for
80 overlaps beyond the correlation itself.
81- **onset_knn**: detection on a smoothed energy envelope, classification
82 by distance-weighted k-nearest-neighbours over spectral shape features
83 (band-energy ratios, centroid, zero-crossing rate). Levels are used only
e06c5e1Default to original event volumes; rename gain knob to level variation with an explanatory tooltipJeremy Magland 84 as ratios, so level variation, when enabled, carries no information.
21abc12audio-spike-sorting: benchmark framework for audio event detection and classificationJeremy Magland 85- **naive_threshold**: an amplitude threshold and a classifier that looks
86 only at event duration. A deliberately weak floor; a real method that
87 fails to clear it is in trouble.
e06c5e1Default to original event volumes; rename gain knob to level variation with an explanatory tooltipJeremy Magland 89On the synthetic demo data (three types, 20% overlap, 12 dB of level
90variation), the baselines land around 0.6 to 0.8 accuracy, with the
91losses concentrated in the overlapping events. None of the built-ins attempts overlap resolution
21abc12audio-spike-sorting: benchmark framework for audio event detection and classificationJeremy Magland 92(subtraction or joint fitting), which is exactly the direction candidate
93algorithms are expected to explore.
95## Running it
97```bash
98npm install
99npm run dev # the app, at the printed URL
100npm test # node end-to-end test of the pipeline (no browser)
101npm run build # typecheck and bundle
102```
104"Load demo data" creates three synthetic event types so the whole
105pipeline can be tried without a microphone. Projects persist in the
106browser (clip audio in IndexedDB, everything else in localStorage) and
107round-trip through Export/Import as a single JSON file, so a recorded
108snippet library can be shared or committed.
110## Notes and limitations
112- The internal rate is 16 kHz mono; microphone input is downsampled on
113 capture with a box average, which is a crude anti-alias filter. Events
114 whose identity lives above 8 kHz will lose that information.
115- Synthesis adds no background noise; the noise floor is whatever the
116 recordings carry. A noise knob is a natural next step, as is an option
117 to place events over a recorded room-tone bed.
118- The matching is greedy rather than globally optimal; with a 20 ms
119 tolerance and events at realistic densities the difference is
120 negligible, but it is not the Hungarian algorithm.
121- Algorithm execution is single-threaded per run, so per-algorithm
122 timings are comparable but a benchmark with many runs takes their sum.