1# instrawave-compose
3Compose rhythms with your voice, or with anything else you can do near a
4microphone. You teach the app a handful of named *voices* by example (say
5"doo" five times, tap the desk five times), then record a *take* in one
6pass: "doo baa doo-doo doo". The take is segmented into hits, each hit is
7classified back onto the voices, and the result is a timeline of timed,
8weighted events that renders to audio through an *instrument* bound to
9each voice.
11A limitation of most rhythm entry tools is the keyboard: tapping a pad or
12clicking a grid discards the timing and dynamics a person can produce
13vocally without effort. Here the performed take is the source of truth.
14Times are kept exactly as performed (there is no quantization grid), and
15each hit's measured loudness becomes the note's strength, taken relative
16to the loudest hit in the take and compressed so soft hits stay audible
17while the ordering is preserved.
19## Voices, takes, and the timeline
21A voice is a named class of sound plus its training examples. Recording
22into a voice runs an onset detector (spectral flux against a local mean,
23with a sensitivity control, since a soft syllable and a sharp tap sit at
24very different flux levels) and opens the clip in an editor: the detected
25hits are markers on the waveform, auditioned by click, selected in groups
26by dragging or shift-clicking, and excluded with a button or the X key, so
27a cough or a stray detection is visible rather than silently poisoning the
28label. Clips persist (audio in IndexedDB, detections and features in
29localStorage) and reopen in the same editor later; deleting a clip removes
30its hits from the voice. Each committed hit is stored as a small feature
31vector: spectral centroid and rolloff, band energy ratios, zero-crossing
32rate, attack and decay times. Loudness is deliberately kept out of this
33vector; it is expression, not identity, so playing softly does not change
34which voice a hit lands on.
36A take is detected and classified by a MATLAB-syntax algorithm executed
37in the browser by [numbl](https://numbl.org), with the same interface the
38[audio-spike-sorting](../audio-spike-sorting) benchmark uses
39(`[times, labels] = name(x, fs, training)`, everything at 16 kHz), so an
40algorithm developed and scored against that benchmark drops in here
41unchanged. The default is silence_gap_knn from the benchmark, with one
42adaptation: its segmentation threshold is raised from a fraction of the
43recording peak to a robust noise-floor estimate, since a microphone's
44background is never digitally silent. A classifier selector in the take
45bar switches between the default (source editable, resettable), a custom
46numbl algorithm pasted into the same editor, and the original in-app
47method: spectral-flux onset detection plus distance-weighted nearest
48neighbours over the pooled per-hit feature vectors. The training the
49numbl algorithms see is the raw snippets cut from the voices' clips
50(20 ms before each kept onset to 180 ms after); hits learned from takes
51carry only features, so they inform the built-in kNN but not the numbl
52path.
54The timeline shows one lane per voice with a marker per hit, bar height
55showing strength. The classifier will sometimes be wrong, and the fix is
56one gesture: drag the marker to the right lane. "Learn" folds the
57corrected hit back into that voice's feature training set, so the
58built-in kNN improves with use; there is no model to refit.
60Everything persists in the browser: voices and their training, the take
61(audio and edited events), per-instrument geometry, slider positions, and
62the bakes themselves, which round-trip through IndexedDB so a reload finds
63them stored and loads one the first time it is needed rather than paying
64the precompute again. During playback each hit flashes as the playhead
65crosses it, ring size following the hit's strength, so the dynamics are
66visible in motion as well as in the bar heights.
68## Rendering
70The seam is `src/render/types.ts`: events in, audio out. Each voice is
71bound either to a placeholder WebAudio synth preset (kick, snare, hat,
72tom) or to an [instrawave](../instrawave) instrument (drum, dulcimer,
73chime) with per-voice tuning overrides, edited as sliders under the voice.
74Playback touches the GPU only when at least one bound instrument is
75instrawave; a purely synth arrangement never asks for a device.
77Each instrument carries one set of geometry and accuracy values, edited
78in the instrument panel that expands inline under a voice row bound to it:
79instrawave's own raymarched scene view alongside the geometry and accuracy
80sliders, so the staircased sound hole or the microphone position is seen
81exactly as the solver sees it. The bake state, the Precompute button, and
82its progress sit inline on the same rows. Geometry belongs to the
83instrument rather than the voice because it is what the shared bake
84depends on; each instrument therefore has exactly one bake, computed on
85demand, and rows sharing an instrument show the same state. A Hear button
86in each
87voice's tuning panel renders a single hit at the current tuning from that
88bake, so tuning is auditioned without recording anything. All per-voice
89character travels as tuning, which is free: every voice bound to the same
90instrument shares its bake, and one render call carries all their events,
91each hit an event `{t, set: {...tuning, vel}}` with the voice's overrides
92and its strength scaling the excitation (`vel`, or `amp` for the plucked
93dulcimer; the equations are linear, so this is a pure scale). Traces come
94back at the solver rate, are resampled by linear interpolation, levelled
95per instrument (a pressure trace carries no absolute scale worth keeping
96against a synth's), and mixed; relative volumes within one instrument stay
97physical.
99Rendered tracks are cached per voice, in memory and in IndexedDB, each
100stamped with exactly what it was rendered from: its events, tuning, gains,
101instrument, and the bake key. Editing one voice therefore re-renders only
102that voice's track, an unchanged take replays for free across reloads, and
103a stored track plays without its bake even being loaded. Each track runs
104to its own last event plus its longest decay, so its signature cannot be
105perturbed by hits on other lanes. Each voice also carries a playback
106volume slider; volume is deliberately absent from the track signature and
107scales the finished track at mix time, so moving it never renders
108anything. Save .wav downloads the current mix as 16-bit PCM at 44.1 kHz,
109rendered through the same caches.
111Loudness to excitation strength is the only feature mapping for now.
112Duration to decay, vocal pitch to drum pitch, and brightness to strike
113position are natural next steps and the event structure leaves room for
114them.
116## Honest limitations
118Onset detection is offline and frame-based; timing is good to a few
119milliseconds on sharp attacks and worse on breathy ones. The classifier is
120nearest-neighbour over hand-picked features, which separates a dozen
121distinct sounds well but has no notion of a hit it has never seen; an
122unfamiliar sound lands on whatever is closest. Overlapping hits closer
123than 50 ms merge into one. The browser's microphone processing is asked to
124stand down (automatic gain control would rewrite the very dynamics being
125tracked), but a system-level AGC outside the browser will still flatten
126them. A stored bake is a couple of hundred MB in IndexedDB for the drum;
127browsers may evict it under storage pressure, in which case the precompute
128(about 8 s for the drum on the development machine) simply runs again. A
129hit learned from a take stores
130only its features, so it cannot be re-auditioned later; clips can. A
131multi-event instrawave render costs one synthesis pass per event over the
132whole trace, so a very dense take renders slowly.
134## Running it
136```
137npm install # expects ../instrawave alongside, per the vault layout
138npm run dev # the page: voices, take, timeline, playback
139npm test # onset, feature, classifier checks on synthetic audio
140npm run build
141```