/ proof-of-concept / audio-spike-sorting
Sign in
proof-of-concept / audio-spike-sorting
32dec4f
Branches
main
audio-spike-sorting / README.md
7.7 KBPreviewCodeBlameHistoryRaw

audio-spike-sorting#

A benchmark framework for the detection and classification of short audio events, in the spirit of spike sorting. You record hundreds of examples of each event type (tapping the table, tapping the floor, tapping with a spoon), cleanly separated so the examples are unambiguous. The framework then synthesizes recordings in which known events are placed at known times, possibly overlapping, hands each candidate algorithm a small labeled training set, and scores the algorithm's output against the exact ground truth. Candidate algorithms are written in MATLAB syntax and executed in the browser by numbl; everything, from recording through scoring and visual exploration, runs on one page with nothing to install.

A limitation of evaluating event-detection methods on real recordings is that the ground truth is itself the product of some annotation process, with its own errors and biases. Synthesis from recorded examples avoids this: the event times and labels are correct by construction, while the audio itself remains real. The trade-off is that a synthesized mixture is not a real performance; in particular, overlapping events here are sums of independently recorded events, which ignores any physical interaction between them.

How a benchmark run works#

  1. Snippets. Each recorded clip is segmented by an onset detector (spectral flux, with an adjustable sensitivity), and the detected events stay editable: click to audition, exclude mistakes. Every kept onset is refined to the foot of its energy rise and a fixed window (default 20 ms before, 180 ms after) is cut around it. This refined onset is the project's single definition of event time, used both for training snippets and for ground truth, so all algorithms are scored against one consistent convention.
  2. Split. For each run, every type's snippets are randomly divided into a training set (default 30%) given to the algorithm, and a placement pool used to build the dataset. An algorithm is therefore always classifying events it has not literally seen.
  3. Synthesis. Events from the placement pool are placed on a silent timeline at their recorded volumes. A level-variation knob can optionally scale each placed event by a random gain drawn between 0 dB and a floor (say -12 dB), so that loudness carries no information about identity; at the default of 0 dB no adjustment is made. An overlap knob in [0, 1] sets the fraction of events deliberately placed inside another event's span; the rest keep at least one snippet length of separation. Run r uses seed + r, so the whole benchmark is reproducible from the configuration.
  4. Execution. Each algorithm is a .m file defining function [times, labels] = name(x, fs, training), where training carries the per-type snippet matrices, the type names, and the onset position within a snippet. numbl runs it in a web worker; the sources are editable in place on the page.
  5. Scoring. Detections are matched to ground truth greedily by time difference within a tolerance (default 20 ms), ignoring labels. A matched pair with the right label is correct, with the wrong label a misclassification; unmatched truth events are misses and unmatched detections false positives. The summary accuracy is correct / (correct + misclassified + missed + false positives), reported as mean and standard deviation over runs, alongside a confusion matrix per algorithm with an extra column for misses and an extra row for false positives.

A completed benchmark can be downloaded as a .zip for offline algorithm development: every run's recording as a float32 WAV, its ground truth as a CSV, the exact training snippets it handed to the algorithms, the algorithm sources, and an evaluation harness (evaluate.m plus tools/) that the numbl CLI executes with npx numbl run evaluate.m, printing the same scores as the browser. The archive's README documents the file formats and the algorithm interface, so an algorithm developed against it drops straight back into the app.

The explorer shows any synthesized recording with the ground truth lanes above the waveform and one algorithm's detections below it. A correct detection is a filled marker in its type's color, a misclassification is ringed in red, a false positive is a red cross, and a missed truth event is a hollow circle. The wheel zooms, dragging pans, and a click auditions the audio at that time. The whole recording can also be played through, with a playhead tracking the sound: shift-drag selects an interval and the play button then plays just that selection, which is the quickest way to listen closely to a cluster of errors; a shift-click clears it.

Baseline algorithms#

Four built-ins seed the comparison and can be edited or reset in place:

On the synthetic demo data (three types, 20% overlap, 12 dB of level variation), the baselines land around 0.6 to 0.8 accuracy, with the losses concentrated in the overlapping events. None of the built-ins attempts overlap resolution (subtraction or joint fitting), which is exactly the direction candidate algorithms are expected to explore.

Running it#

npm install
npm run dev        # the app, at the printed URL
npm test           # node end-to-end test of the pipeline (no browser)
npm run build      # typecheck and bundle

"Load demo data" creates three synthetic event types so the whole pipeline can be tried without a microphone.

npm run test:offline exercises the dataset download end to end: it builds the archive's files in node, runs the shipped evaluate.m through the numbl CLI, and requires the printed per-run accuracies to match the in-process scores exactly. Projects persist in the browser (clip audio in IndexedDB, everything else in localStorage) and round-trip through Export/Import as a single JSON file, so a recorded snippet library can be shared or committed.

Notes and limitations#

↑↓ move↵openescclose