1# instrawave-compose
3Compose rhythms with your voice, or with anything else you can do near a
4microphone. You teach the app a handful of named *voices* by example (say
5"doo" five times, tap the desk five times), then record a *take* in one
6pass: "doo baa doo-doo doo". The take is segmented into hits, each hit is
7classified back onto the voices, and the result is a timeline of timed,
8weighted events that renders to audio through an *instrument* bound to
9each voice.
11A limitation of most rhythm entry tools is the keyboard: tapping a pad or
12clicking a grid discards the timing and dynamics a person can produce
13vocally without effort. Here the performed take is the source of truth.
14Times are kept exactly as performed (there is no quantization grid), and
15each hit's measured loudness becomes the note's strength, taken relative
16to the loudest hit in the take and compressed so soft hits stay audible
17while the ordering is preserved.
19## Voices, takes, and the timeline
21A voice is a named class of sound plus its training examples. Recording
22into a voice runs an onset detector (spectral flux against a local mean,
23with a sensitivity control, since a soft syllable and a sharp tap sit at
24very different flux levels) and shows the detected hits as markers on the
25waveform; each can be auditioned and excluded before the batch is
26committed, so a cough or a stray detection is visible rather than silently
27poisoning the label. Each committed hit is stored as a small feature
28vector: spectral centroid and rolloff, band energy ratios, zero-crossing
29rate, attack and decay times. Loudness is deliberately kept out of this
30vector; it is expression, not identity, so playing softly does not change
31which voice a hit lands on.
33A take goes through the same detector, and each hit is classified by
34distance-weighted nearest neighbours over the pooled training set,
35z-scored per dimension. The timeline shows one lane per voice with a
36marker per hit, bar height showing strength. The classifier will sometimes
37be wrong, and the fix is one gesture: drag the marker to the right lane.
38"Learn" folds the corrected hit back into that voice's training set, so
39the labelling improves with use; there is no model to refit.
41Voices and their training persist in localStorage. Takes do not; the take
42is a performance, and re-recording one is the cheap operation.
44## Rendering
46The seam is `src/render/types.ts`: events in, audio out. The current
47renderer is a set of placeholder WebAudio drum synths (kick, snare, hat,
48tom) so the whole capture-classify-arrange loop is playable end to end.
49The intended renderer is [instrawave](../instrawave), where a binding
50becomes an instrument file plus parameter values and each hit becomes a
51stimulus event `{t, set: {vel: ...}}`. Because an instrawave bake depends
52only on geometry and accuracy, voices that share a bake and differ only in
53tuning (a low, wide, centre-struck drum against a high edge strike) render
54in a single pass; the renderer will group events by bake key and mix one
55trace per group. Loudness-to-velocity is the only feature mapping for now.
56Duration to decay, vocal pitch to drum pitch, and brightness to strike
57position are natural next steps and the event structure leaves room for
58them.
60## Honest limitations
62Onset detection is offline and frame-based; timing is good to a few
63milliseconds on sharp attacks and worse on breathy ones. The classifier is
64nearest-neighbour over hand-picked features, which separates a dozen
65distinct sounds well but has no notion of a hit it has never seen; an
66unfamiliar sound lands on whatever is closest. Overlapping hits closer
67than 50 ms merge into one. The browser's microphone processing is asked to
68stand down (automatic gain control would rewrite the very dynamics being
69tracked), but a system-level AGC outside the browser will still flatten
70them. Training audio is not persisted, only its features, so trained hits
71cannot be re-auditioned after a reload.
73## Running it
75```
76npm install # expects ../instrawave alongside, per the vault layout
77npm run dev # the page: voices, take, timeline, playback
78npm test # onset, feature, classifier checks on synthetic audio
79npm run build
80```