1# instrawave-compose
3Compose rhythms with your voice, or with anything else you can do near a
4microphone. You teach the app a handful of named *voices* by example (say
5"doo" five times, tap the desk five times), then record a *take* in one
6pass: "doo baa doo-doo doo". The take is segmented into hits, each hit is
7classified back onto the voices, and the result is a timeline of timed,
8weighted events that renders to audio through an *instrument* bound to
9each voice.
11A limitation of most rhythm entry tools is the keyboard: tapping a pad or
12clicking a grid discards the timing and dynamics a person can produce
13vocally without effort. Here the performed take is the source of truth.
14Times are kept exactly as performed (there is no quantization grid), and
15each hit's measured loudness becomes the note's strength, taken relative
16to the loudest hit in the take and compressed so soft hits stay audible
17while the ordering is preserved.
19## Voices, takes, and the timeline
21A voice is a named class of sound plus its training examples. Recording
22into a voice runs an onset detector (spectral flux against a local mean,
23with a sensitivity control, since a soft syllable and a sharp tap sit at
24very different flux levels) and opens the clip in an editor: the detected
25hits are markers on the waveform, auditioned by click, selected in groups
26by dragging or shift-clicking, and excluded with a button or the X key, so
27a cough or a stray detection is visible rather than silently poisoning the
28label. Clips persist (audio in IndexedDB, detections and features in
29localStorage) and reopen in the same editor later; deleting a clip removes
30its hits from the voice. Each committed hit is stored as a small feature
31vector: spectral centroid and rolloff, band energy ratios, zero-crossing
32rate, attack and decay times. Loudness is deliberately kept out of this
33vector; it is expression, not identity, so playing softly does not change
34which voice a hit lands on.
36A take goes through the same detector, and each hit is classified by
37distance-weighted nearest neighbours over the pooled training set,
38z-scored per dimension. The timeline shows one lane per voice with a
39marker per hit, bar height showing strength. The classifier will sometimes
40be wrong, and the fix is one gesture: drag the marker to the right lane.
41"Learn" folds the corrected hit back into that voice's training set, so
42the labelling improves with use; there is no model to refit.
44Takes are not persisted; the take is a performance, and re-recording one
45is the cheap operation. During playback each hit flashes as the playhead
46crosses it, ring size following the hit's strength, so the dynamics are
47visible in motion as well as in the bar heights.
49## Rendering
51The seam is `src/render/types.ts`: events in, audio out. Each voice is
52bound either to a placeholder WebAudio synth preset (kick, snare, hat,
53tom) or to an [instrawave](../instrawave) instrument (drum, dulcimer,
54chime) with per-voice tuning overrides, edited as sliders under the voice.
55Playback touches the GPU only when at least one bound instrument is
56instrawave; a purely synth arrangement never asks for a device.
58The instrawave instruments are held at their default geometry and
59accuracy, so each has exactly one bake, computed on demand from the bake
60bar (with progress and a cancel) and kept for the session. All per-voice
61character travels as tuning, which is free: every voice bound to the same
62instrument shares its bake, and one render call carries all their events,
63each hit an event `{t, set: {...tuning, vel}}` with the voice's overrides
64and its strength scaling the excitation (`vel`, or `amp` for the plucked
65dulcimer; the equations are linear, so this is a pure scale). Traces come
66back at the solver rate, are resampled by linear interpolation, levelled
67per instrument (a pressure trace carries no absolute scale worth keeping
68against a synth's), and mixed; relative volumes within one instrument stay
69physical. Mixes are cached against the exact events and bindings, so
70replaying an unchanged take is free.
72Loudness to excitation strength is the only feature mapping for now.
73Duration to decay, vocal pitch to drum pitch, and brightness to strike
74position are natural next steps and the event structure leaves room for
75them.
77## Honest limitations
79Onset detection is offline and frame-based; timing is good to a few
80milliseconds on sharp attacks and worse on breathy ones. The classifier is
81nearest-neighbour over hand-picked features, which separates a dozen
82distinct sounds well but has no notion of a hit it has never seen; an
83unfamiliar sound lands on whatever is closest. Overlapping hits closer
84than 50 ms merge into one. The browser's microphone processing is asked to
85stand down (automatic gain control would rewrite the very dynamics being
86tracked), but a system-level AGC outside the browser will still flatten
87them. Bakes are not persisted (the drum's is a couple of hundred MB), so
88the first instrawave playback of a session pays the precompute, about 8 s
89for the drum on the development machine. A hit learned from a take stores
90only its features, so it cannot be re-auditioned later; clips can. A
91multi-event instrawave render costs one synthesis pass per event over the
92whole trace, so a very dense take renders slowly.
94## Running it
96```
97npm install # expects ../instrawave alongside, per the vault layout
98npm run dev # the page: voices, take, timeline, playback
99npm test # onset, feature, classifier checks on synthetic audio
100npm run build
101```