instrawave-compose#

Compose rhythms with your voice, or with anything else you can do near a microphone. You teach the app a handful of named voices by example (say "doo" five times, tap the desk five times), then record a take in one pass: "doo baa doo-doo doo". The take is segmented into hits, each hit is classified back onto the voices, and the result is a timeline of timed, weighted events that renders to audio through an instrument bound to each voice.

A limitation of most rhythm entry tools is the keyboard: tapping a pad or clicking a grid discards the timing and dynamics a person can produce vocally without effort. Here the performed take is the source of truth. Times are kept exactly as performed (there is no quantization grid), and each hit's measured loudness becomes the note's strength, taken relative to the loudest hit in the take and compressed so soft hits stay audible while the ordering is preserved.

Voices, takes, and the timeline#

A voice is a named class of sound plus its training examples. Recording into a voice runs an onset detector (spectral flux against a local mean, with a sensitivity control, since a soft syllable and a sharp tap sit at very different flux levels) and opens the clip in an editor: the detected hits are markers on the waveform, auditioned by click, selected in groups by dragging or shift-clicking, and excluded with a button or the X key, so a cough or a stray detection is visible rather than silently poisoning the label. Clips persist (audio in IndexedDB, detections and features in localStorage) and reopen in the same editor later; deleting a clip removes its hits from the voice. Each committed hit is stored as a small feature vector: spectral centroid and rolloff, band energy ratios, zero-crossing rate, attack and decay times. Loudness is deliberately kept out of this vector; it is expression, not identity, so playing softly does not change which voice a hit lands on.

A take is detected and classified by a MATLAB-syntax algorithm executed in the browser by numbl, with the same interface the audio-spike-sorting benchmark uses ([times, labels] = name(x, fs, training), everything at 16 kHz), so an algorithm developed and scored against that benchmark drops in here unchanged. The default is silence_gap_knn from the benchmark, with one adaptation: its segmentation threshold is raised from a fraction of the recording peak to a robust noise-floor estimate, since a microphone's background is never digitally silent. A classifier selector in the take bar switches between the default (source editable, resettable), a custom numbl algorithm pasted into the same editor, and the original in-app method: spectral-flux onset detection plus distance-weighted nearest neighbours over the pooled per-hit feature vectors. The training the numbl algorithms see is the raw snippets cut from the voices' clips (20 ms before each kept onset to 180 ms after); hits learned from takes carry only features, so they inform the built-in kNN but not the numbl path.

The timeline shows one lane per voice with a marker per hit, bar height showing strength. The classifier will sometimes be wrong, and the fix is one gesture: drag the marker to the right lane. "Learn" folds the corrected hit back into that voice's feature training set, so the built-in kNN improves with use; there is no model to refit.

Everything persists in the browser: voices and their training, the take (audio and edited events), per-instrument geometry, slider positions, and the bakes themselves, which round-trip through IndexedDB so a reload finds them stored and loads one the first time it is needed rather than paying the precompute again. During playback each hit flashes as the playhead crosses it, ring size following the hit's strength, so the dynamics are visible in motion as well as in the bar heights.

Rendering#

The seam is src/render/types.ts: events in, audio out. Each voice is bound either to a placeholder WebAudio synth preset (kick, snare, hat, tom) or to an instrawave instrument (drum, dulcimer, chime) with per-voice tuning overrides, edited as sliders under the voice. Playback touches the GPU only when at least one bound instrument is instrawave; a purely synth arrangement never asks for a device.

Each instrument carries one set of geometry and accuracy values, edited in the instrument panel that expands inline under a voice row bound to it: instrawave's own raymarched scene view alongside the geometry and accuracy sliders, so the staircased sound hole or the microphone position is seen exactly as the solver sees it. The bake state, the Precompute button, and its progress sit inline on the same rows. Geometry belongs to the instrument rather than the voice because it is what the shared bake depends on; each instrument therefore has exactly one bake, computed on demand, and rows sharing an instrument show the same state. A Hear button in each voice's tuning panel renders a single hit at the current tuning from that bake, so tuning is auditioned without recording anything. All per-voice character travels as tuning, which is free: every voice bound to the same instrument shares its bake, and one render call carries all their events, each hit an event {t, set: {...tuning, vel}} with the voice's overrides and its strength scaling the excitation (vel, or amp for the plucked dulcimer; the equations are linear, so this is a pure scale). Traces come back at the solver rate, are resampled by linear interpolation, levelled per instrument (a pressure trace carries no absolute scale worth keeping against a synth's), and mixed; relative volumes within one instrument stay physical.

Rendered tracks are cached per voice, in memory and in IndexedDB, each stamped with exactly what it was rendered from: its events, tuning, gains, instrument, and the bake key. Editing one voice therefore re-renders only that voice's track, an unchanged take replays for free across reloads, and a stored track plays without its bake even being loaded. Each track runs to its own last event plus its longest decay, so its signature cannot be perturbed by hits on other lanes. Each voice also carries a playback volume slider; volume is deliberately absent from the track signature and scales the finished track at mix time, so moving it never renders anything. Save .wav downloads the current mix as 16-bit PCM at 44.1 kHz, rendered through the same caches.

Loudness to excitation strength is the only feature mapping for now. Duration to decay, vocal pitch to drum pitch, and brightness to strike position are natural next steps and the event structure leaves room for them.

Honest limitations#

Onset detection is offline and frame-based; timing is good to a few milliseconds on sharp attacks and worse on breathy ones. The classifier is nearest-neighbour over hand-picked features, which separates a dozen distinct sounds well but has no notion of a hit it has never seen; an unfamiliar sound lands on whatever is closest. Overlapping hits closer than 50 ms merge into one. The browser's microphone processing is asked to stand down (automatic gain control would rewrite the very dynamics being tracked), but a system-level AGC outside the browser will still flatten them. A stored bake is a couple of hundred MB in IndexedDB for the drum; browsers may evict it under storage pressure, in which case the precompute (about 8 s for the drum on the development machine) simply runs again. A hit learned from a take stores only its features, so it cannot be re-auditioned later; clips can. A multi-event instrawave render costs one synthesis pass per event over the whole trace, so a very dense take renders slowly.

Running it#

npm install        # expects ../instrawave alongside, per the vault layout
npm run dev        # the page: voices, take, timeline, playback
npm test           # onset, feature, classifier checks on synthetic audio
npm run build