Voice-driven rhythm capture: voices trained by example, takes classified onto them, synth playback
Onset detection by spectral flux, per-hit identity features with loudness
kept separate as expression, kNN classification with one-gesture
correction on the timeline, and playback through a renderer seam that the
instrawave event-list renderer will implement next.
18 changed files+3935−0
.gitignoreadded+2−0View file
@@ -0,0 +1,2 @@
1+node_modules/
2+dist/
README.mdadded+80−0View file
@@ -0,0 +1,80 @@
1+# instrawave-compose
2+
3+Compose rhythms with your voice, or with anything else you can do near a
4+microphone. You teach the app a handful of named *voices* by example (say
5+"doo" five times, tap the desk five times), then record a *take* in one
6+pass: "doo baa doo-doo doo". The take is segmented into hits, each hit is
7+classified back onto the voices, and the result is a timeline of timed,
8+weighted events that renders to audio through an *instrument* bound to
9+each voice.
10+
11+A limitation of most rhythm entry tools is the keyboard: tapping a pad or
12+clicking a grid discards the timing and dynamics a person can produce
13+vocally without effort. Here the performed take is the source of truth.
14+Times are kept exactly as performed (there is no quantization grid), and
15+each hit's measured loudness becomes the note's strength, taken relative
16+to the loudest hit in the take and compressed so soft hits stay audible
17+while the ordering is preserved.
18+
19+## Voices, takes, and the timeline
20+
21+A voice is a named class of sound plus its training examples. Recording
22+into a voice runs an onset detector (spectral flux against a local mean,
23+with a sensitivity control, since a soft syllable and a sharp tap sit at
24+very different flux levels) and shows the detected hits as markers on the
25+waveform; each can be auditioned and excluded before the batch is
26+committed, so a cough or a stray detection is visible rather than silently
27+poisoning the label. Each committed hit is stored as a small feature
28+vector: spectral centroid and rolloff, band energy ratios, zero-crossing
29+rate, attack and decay times. Loudness is deliberately kept out of this
30+vector; it is expression, not identity, so playing softly does not change
31+which voice a hit lands on.
32+
33+A take goes through the same detector, and each hit is classified by
34+distance-weighted nearest neighbours over the pooled training set,
35+z-scored per dimension. The timeline shows one lane per voice with a
36+marker per hit, bar height showing strength. The classifier will sometimes
37+be wrong, and the fix is one gesture: drag the marker to the right lane.
38+"Learn" folds the corrected hit back into that voice's training set, so
39+the labelling improves with use; there is no model to refit.
40+
41+Voices and their training persist in localStorage. Takes do not; the take
42+is a performance, and re-recording one is the cheap operation.
43+
44+## Rendering
45+
46+The seam is `src/render/types.ts`: events in, audio out. The current
47+renderer is a set of placeholder WebAudio drum synths (kick, snare, hat,
48+tom) so the whole capture-classify-arrange loop is playable end to end.
49+The intended renderer is [instrawave](../instrawave), where a binding
50+becomes an instrument file plus parameter values and each hit becomes a
51+stimulus event `{t, set: {vel: ...}}`. Because an instrawave bake depends
52+only on geometry and accuracy, voices that share a bake and differ only in
53+tuning (a low, wide, centre-struck drum against a high edge strike) render
54+in a single pass; the renderer will group events by bake key and mix one
55+trace per group. Loudness-to-velocity is the only feature mapping for now.
56+Duration to decay, vocal pitch to drum pitch, and brightness to strike
57+position are natural next steps and the event structure leaves room for
58+them.
59+
60+## Honest limitations
61+
62+Onset detection is offline and frame-based; timing is good to a few
63+milliseconds on sharp attacks and worse on breathy ones. The classifier is
64+nearest-neighbour over hand-picked features, which separates a dozen
65+distinct sounds well but has no notion of a hit it has never seen; an
66+unfamiliar sound lands on whatever is closest. Overlapping hits closer
67+than 50 ms merge into one. The browser's microphone processing is asked to
68+stand down (automatic gain control would rewrite the very dynamics being
69+tracked), but a system-level AGC outside the browser will still flatten
70+them. Training audio is not persisted, only its features, so trained hits
71+cannot be re-auditioned after a reload.
72+
73+## Running it
74+
75+```
76+npm install # expects ../instrawave alongside, per the vault layout
77+npm run dev # the page: voices, take, timeline, playback
78+npm test # onset, feature, classifier checks on synthetic audio
79+npm run build
80+```
index.htmladded+97−0View file
@@ -0,0 +1,97 @@
1+<!doctype html>
2+<html lang="en">
3+ <head>
4+ <meta charset="utf-8" />
5+ <meta name="viewport" content="width=device-width, initial-scale=1" />
6+ <title>instrawave-compose</title>
7+ <style>
8+ :root {
9+ color-scheme: light dark;
10+ font-family: system-ui, sans-serif;
11+ }
12+ body {
13+ margin: 0 auto;
14+ max-width: 72rem;
15+ padding: 1rem 1.5rem 4rem;
16+ line-height: 1.45;
17+ }
18+ h1 { font-size: 1.3rem; margin: 0.3rem 0 0; }
19+ h1 + p { margin-top: 0.2rem; opacity: 0.75; }
20+ h2 { font-size: 0.95rem; margin: 1.4rem 0 0.4rem; }
21+ .cost { font-weight: normal; opacity: 0.6; font-size: 0.8rem; }
22+ .bar { display: flex; gap: 0.6rem; align-items: center; flex-wrap: wrap; margin: 0.5rem 0; }
23+ button { font: inherit; padding: 0.3rem 0.8rem; }
24+ button:disabled { opacity: 0.5; }
25+ select, input[type='text'] { font: inherit; }
26+ input[type='range'] { width: 9rem; }
27+ .voice-row {
28+ display: flex; gap: 0.6rem; align-items: center; flex-wrap: wrap;
29+ padding: 0.3rem 0.4rem; border-radius: 6px; font-size: 0.9rem;
30+ }
31+ .voice-row:nth-child(odd) { background: rgba(128, 128, 128, 0.09); }
32+ .voice-row .swatch { width: 0.8rem; height: 0.8rem; border-radius: 3px; flex: none; }
33+ .voice-row input.name { width: 8rem; border: none; background: transparent; font-weight: 600; }
34+ .voice-row .count { opacity: 0.65; font-size: 0.8rem; min-width: 5.5rem; }
35+ .voice-row .spacer { flex: 1; }
36+ .rec { color: #d33; }
37+ canvas.wave { width: 100%; height: 110px; border-radius: 6px; display: block; touch-action: none; }
38+ canvas.tl { width: 100%; border-radius: 6px; display: block; touch-action: none; }
39+ .meter { width: 7rem; height: 0.55rem; border-radius: 3px; background: rgba(128,128,128,0.25); overflow: hidden; }
40+ .meter div { height: 100%; width: 0; background: #3dbb6e; }
41+ #capture { border: 1px solid rgba(128,128,128,0.35); border-radius: 8px; padding: 0.6rem 0.8rem; margin-top: 0.8rem; }
42+ #status { font-size: 0.85rem; opacity: 0.8; min-height: 1.3em; }
43+ .hint { font-size: 0.8rem; opacity: 0.6; }
44+ label { display: inline-flex; gap: 0.35rem; align-items: center; }
45+ </style>
46+ </head>
47+ <body>
48+ <h1>instrawave-compose</h1>
49+ <p>
50+ Teach it your sounds as named voices, then perform a rhythm in one take;
51+ hits are classified back onto the voices and rendered on instruments.
52+ </p>
53+
54+ <h2>Voices <span class="cost">record several examples of each sound: a syllable, a tap, anything repeatable</span></h2>
55+ <div id="voice-list"></div>
56+ <div class="bar">
57+ <input id="voice-name" type="text" placeholder="new voice name (e.g. doo)" />
58+ <button id="add-voice">Add voice</button>
59+ <span class="spacer" style="flex:1"></span>
60+ <button id="reset-all">Forget everything</button>
61+ </div>
62+
63+ <div id="capture" hidden>
64+ <div class="bar">
65+ <strong id="capture-title"></strong>
66+ <span class="hint">click a marker to audition; exclude anything that is not the sound</span>
67+ </div>
68+ <canvas id="capture-wave" class="wave"></canvas>
69+ <div class="bar">
70+ <label>sensitivity <input id="capture-sens" type="range" min="0" max="1" step="0.01" value="0.6" /></label>
71+ <button id="capture-exclude" disabled>Exclude</button>
72+ <span style="flex:1"></span>
73+ <button id="capture-commit"></button>
74+ <button id="capture-discard">Discard</button>
75+ </div>
76+ </div>
77+
78+ <h2>Take <span class="cost">one pass, exactly as performed; no grid</span></h2>
79+ <div class="bar">
80+ <button id="take-record">● Record take</button>
81+ <div class="meter" id="meter" hidden><div></div></div>
82+ <label>sensitivity <input id="take-sens" type="range" min="0" max="1" step="0.01" value="0.6" /></label>
83+ <button id="take-redetect" disabled>Re-detect & classify</button>
84+ <span style="flex:1"></span>
85+ <button id="take-play" disabled>Play</button>
86+ </div>
87+ <canvas id="timeline" class="tl"></canvas>
88+ <div class="bar">
89+ <span class="hint">drag a hit to another lane to reassign · double-click a lane to add a hit · delete key removes</span>
90+ <span style="flex:1"></span>
91+ <button id="hit-learn" disabled>Learn selected into its voice</button>
92+ <button id="hit-delete" disabled>Delete selected</button>
93+ </div>
94+ <p id="status"></p>
95+ <script type="module" src="./src/main.ts"></script>
96+ </body>
97+</html>
package-lock.jsonadded+2361−0View file
This diff is 2,366 lines long and is not shown.
package.jsonadded+24−0View file
@@ -0,0 +1,24 @@
1+{
2+ "name": "instrawave-compose",
3+ "version": "0.1.0",
4+ "description": "Compose rhythms by voice: record syllables or taps, learn them as named voices, transcribe a performed take into timed events, and render the events on instruments",
5+ "type": "module",
6+ "engines": {
7+ "node": ">=22.6"
8+ },
9+ "license": "Apache-2.0",
10+ "scripts": {
11+ "dev": "vite",
12+ "build": "tsc --noEmit && vite build",
13+ "test": "vite-node scripts/test-node.ts"
14+ },
15+ "dependencies": {
16+ "instrawave": "file:../instrawave"
17+ },
18+ "devDependencies": {
19+ "@types/node": "^26.1.1",
20+ "typescript": "^5.5.0",
21+ "vite": "^5.4.0",
22+ "vite-node": "^6.0.0"
23+ }
24+}
scripts/test-node.tsadded+96−0View file
@@ -0,0 +1,96 @@
1+/**
2+ * Checks for the analysis half, run on synthetic audio so no microphone or
3+ * browser is involved: onset detection recovers known hit times, the
4+ * feature extractor separates a low thump from a high tick, and the
5+ * classifier labels held-out hits correctly. Run with `npm test`.
6+ */
7+import { detectOnsets } from '../src/analysis/onsets.ts';
8+import { extractHit } from '../src/analysis/features.ts';
9+import { classify, type TrainingSet } from '../src/analysis/classify.ts';
10+import { gainFromPeakDb } from '../src/render/types.ts';
11+
12+const SR = 48000;
13+let failures = 0;
14+
15+function check(cond: boolean, msg: string): void {
16+ console.log(`${cond ? 'ok ' : 'FAIL'} ${msg}`);
17+ if (!cond) failures++;
18+}
19+
20+/** A decaying sine burst with a touch of noise; freq and decay set its character. */
21+function burst(x: Float32Array, t0: number, freq: number, decay: number, amp: number, seed: number): void {
22+ const s0 = Math.floor(t0 * SR);
23+ const n = Math.min(Math.floor(0.3 * SR), x.length - s0);
24+ let s = seed;
25+ for (let i = 0; i < n; i++) {
26+ s = (s * 1103515245 + 12345) & 0x7fffffff;
27+ const noise = (s / 0x3fffffff - 1) * 0.1;
28+ x[s0 + i] += amp * Math.exp(-i / (decay * SR)) * (Math.sin((2 * Math.PI * freq * i) / SR) + noise);
29+ }
30+}
31+
32+// --- onset detection ---------------------------------------------------------
33+
34+{
35+ const times = [0.2, 0.55, 0.62, 1.1, 1.4];
36+ const x = new Float32Array(Math.floor(2 * SR));
37+ for (const [i, t] of times.entries()) burst(x, t, 200 + 300 * (i % 2), 0.03, 0.5, 7 + i);
38+ const found = detectOnsets(x, SR, 0.6);
39+ check(found.length === times.length, `onsets: found ${found.length} of ${times.length}`);
40+ if (found.length === times.length) {
41+ const worst = Math.max(...times.map((t, i) => Math.abs(found[i] - t)));
42+ check(worst < 0.012, `onsets: worst timing error ${(worst * 1000).toFixed(1)} ms < 12 ms`);
43+ }
44+}
45+
46+// --- quiet hits at low vs high sensitivity -----------------------------------
47+
48+{
49+ const x = new Float32Array(Math.floor(1.5 * SR));
50+ burst(x, 0.2, 250, 0.03, 0.5, 3);
51+ burst(x, 0.7, 250, 0.03, 0.04, 4); // 22 dB quieter
52+ const insensitive = detectOnsets(x, SR, 0.1);
53+ const sensitive = detectOnsets(x, SR, 0.95);
54+ check(sensitive.length >= insensitive.length, `sensitivity: ${sensitive.length} hits at 0.95 >= ${insensitive.length} at 0.1`);
55+ check(sensitive.length === 2, `sensitivity 0.95 finds the quiet hit (${sensitive.length}/2)`);
56+}
57+
58+// --- features + classifier ----------------------------------------------------
59+
60+{
61+ // Two classes: a low, slow thump and a high, fast tick.
62+ const mk = (freq: number, decay: number, amp: number, seed: number) => {
63+ const x = new Float32Array(Math.floor(0.5 * SR));
64+ burst(x, 0.05, freq, decay, amp, seed);
65+ return extractHit(x, SR, Math.floor(0.05 * SR), x.length);
66+ };
67+ const thumps = [0, 1, 2, 3].map((i) => mk(85 + 6 * i, 0.06, 0.4 + 0.1 * i, 11 + i));
68+ const ticks = [0, 1, 2, 3].map((i) => mk(3600 + 250 * i, 0.008, 0.3 + 0.1 * i, 31 + i));
69+
70+ const train: TrainingSet = {
71+ voiceIds: ['thump', 'thump', 'thump', 'tick', 'tick', 'tick'],
72+ vecs: [...thumps.slice(0, 3), ...ticks.slice(0, 3)].map((f) => f.vec),
73+ };
74+ const a = classify(thumps[3].vec, train);
75+ const b = classify(ticks[3].vec, train);
76+ check(a?.voiceId === 'thump', `classify: held-out thump -> ${a?.voiceId} (conf ${a?.confidence.toFixed(2)})`);
77+ check(b?.voiceId === 'tick', `classify: held-out tick -> ${b?.voiceId} (conf ${b?.confidence.toFixed(2)})`);
78+
79+ // Loudness must not leak into identity: a soft thump is still a thump.
80+ const soft = mk(90, 0.06, 0.02, 55);
81+ const c = classify(soft.vec, train);
82+ check(c?.voiceId === 'thump', `classify: 26 dB softer thump -> ${c?.voiceId}`);
83+ check(soft.peakDb < thumps[0].peakDb - 15, `loudness lives in peakDb (${soft.peakDb.toFixed(1)} vs ${thumps[0].peakDb.toFixed(1)} dB)`);
84+}
85+
86+// --- gain mapping --------------------------------------------------------------
87+
88+{
89+ check(Math.abs(gainFromPeakDb(-10, -10) - 1) < 1e-9, 'gain: the loudest hit maps to 1');
90+ const g6 = gainFromPeakDb(-16, -10);
91+ check(g6 > 0.6 && g6 < 0.8, `gain: -6 dB maps to ${g6.toFixed(2)} (compressed, ordering kept)`);
92+ check(gainFromPeakDb(-90, -10) > 0.15, 'gain: very soft hits stay audible');
93+}
94+
95+console.log(failures ? `\n${failures} failure(s)` : '\nall checks passed');
96+process.exit(failures ? 1 : 0);
src/analysis/classify.tsadded+48−0View file
@@ -0,0 +1,48 @@
1+/**
2+ * Nearest-neighbour classification over the identity features. Dimensions
3+ * are z-scored against the pooled training set so no single feature
4+ * dominates by units; the vote is distance-weighted over the k nearest
5+ * training hits. There is no model to fit, so relabelling a hit in the
6+ * timeline and folding it back takes effect on the next take immediately.
7+ */
8+export interface TrainingSet {
9+ voiceIds: string[]; // one per training hit
10+ vecs: number[][]; // matching identity vectors
11+}
12+
13+const K = 3;
14+
15+export function classify(vec: number[], train: TrainingSet): { voiceId: string; confidence: number } | null {
16+ const n = train.vecs.length;
17+ if (n === 0) return null;
18+ const dims = vec.length;
19+
20+ const mean = new Float64Array(dims);
21+ const std = new Float64Array(dims);
22+ for (const v of train.vecs) for (let d = 0; d < dims; d++) mean[d] += v[d];
23+ for (let d = 0; d < dims; d++) mean[d] /= n;
24+ for (const v of train.vecs) for (let d = 0; d < dims; d++) std[d] += (v[d] - mean[d]) ** 2;
25+ for (let d = 0; d < dims; d++) std[d] = Math.sqrt(std[d] / n) || 1;
26+
27+ const dist: { d: number; voiceId: string }[] = [];
28+ for (let i = 0; i < n; i++) {
29+ let s = 0;
30+ for (let d = 0; d < dims; d++) {
31+ const z = (vec[d] - train.vecs[i][d]) / std[d];
32+ s += z * z;
33+ }
34+ dist.push({ d: Math.sqrt(s), voiceId: train.voiceIds[i] });
35+ }
36+ dist.sort((a, b) => a.d - b.d);
37+
38+ const votes = new Map<string, number>();
39+ let totalW = 0;
40+ for (const { d, voiceId } of dist.slice(0, Math.min(K, n))) {
41+ const w = 1 / (d + 1e-3);
42+ votes.set(voiceId, (votes.get(voiceId) ?? 0) + w);
43+ totalW += w;
44+ }
45+ let best = '', bestW = -1;
46+ for (const [id, w] of votes) if (w > bestW) { best = id; bestW = w; }
47+ return { voiceId: best, confidence: bestW / totalW };
48+}
src/analysis/features.tsadded+90−0View file
@@ -0,0 +1,90 @@
1+/**
2+ * Per-hit features. Two roles are kept apart: `peakDb` is expression (how
3+ * hard the hit was; it becomes the note's strength and is deliberately
4+ * excluded from classification, so playing softly does not change which
5+ * voice a hit lands on), and `vec` is identity (spectral shape, attack,
6+ * decay), which is what the classifier compares. Every entry in `vec` is a
7+ * log or a ratio, so distances behave across the huge dynamic ranges
8+ * involved.
9+ */
10+import { powerSpectrum } from '../audio/fft.ts';
11+
12+export interface HitFeatures {
13+ /** Peak short-term RMS over the hit, dBFS. */
14+ peakDb: number;
15+ /** Identity vector; order fixed, see VEC_NAMES. */
16+ vec: number[];
17+}
18+
19+export const VEC_NAMES = [
20+ 'log centroid', 'log rolloff85',
21+ 'band 0-200', 'band 200-800', 'band 800-2500', 'band 2500+',
22+ 'zcr kHz', 'log attack ms', 'log decay ms',
23+];
24+
25+const SPEC_N = 4096;
26+const ENV_WIN = 256;
27+const ENV_HOP = 128;
28+const EPS = 1e-12;
29+
30+/** Extract features for the hit starting at sample s0; s1 bounds it (the
31+ * next onset or the end of the clip). */
32+export function extractHit(x: Float32Array, sr: number, s0: number, s1: number): HitFeatures {
33+ const end = Math.min(x.length, s1, s0 + Math.floor(0.5 * sr));
34+ const seg = x.subarray(s0, Math.max(end, s0 + ENV_WIN));
35+
36+ // Short-term RMS envelope in dB.
37+ const nEnv = Math.max(1, Math.floor((seg.length - ENV_WIN) / ENV_HOP) + 1);
38+ const envDb = new Float32Array(nEnv);
39+ let peakDb = -160, peakI = 0;
40+ for (let i = 0; i < nEnv; i++) {
41+ const off = i * ENV_HOP;
42+ let e = 0;
43+ for (let j = 0; j < ENV_WIN && off + j < seg.length; j++) e += seg[off + j] * seg[off + j];
44+ envDb[i] = 10 * Math.log10(e / ENV_WIN + EPS);
45+ if (envDb[i] > peakDb) { peakDb = envDb[i]; peakI = i; }
46+ }
47+ const attackMs = (peakI * ENV_HOP * 1000) / sr;
48+ let decayI = nEnv - 1;
49+ for (let i = peakI; i < nEnv; i++) if (envDb[i] < peakDb - 25) { decayI = i; break; }
50+ const decayMs = ((decayI - peakI) * ENV_HOP * 1000) / sr;
51+
52+ // Spectral shape of the attack portion (first ~85 ms), one padded FFT.
53+ const spec = powerSpectrum(seg.subarray(0, Math.min(seg.length, SPEC_N)), SPEC_N);
54+ const hzPerBin = sr / SPEC_N;
55+ const kMax = Math.min(spec.length, Math.floor(9000 / hzPerBin));
56+ let total = 0, weighted = 0;
57+ for (let k = 1; k < kMax; k++) { total += spec[k]; weighted += spec[k] * k * hzPerBin; }
58+ const centroid = total > 0 ? weighted / total : 0;
59+ let acc = 0, rolloff = 0;
60+ for (let k = 1; k < kMax; k++) { acc += spec[k]; if (acc >= 0.85 * total) { rolloff = k * hzPerBin; break; } }
61+
62+ // Band energies as log ratios with the mean removed, so overall level
63+ // cancels and only the balance between bands remains.
64+ const edges = [0, 200, 800, 2500, 9000];
65+ const bands: number[] = [];
66+ for (let b = 0; b < 4; b++) {
67+ const k0 = Math.max(1, Math.floor(edges[b] / hzPerBin));
68+ const k1 = Math.min(spec.length, Math.floor(edges[b + 1] / hzPerBin));
69+ let e = 0;
70+ for (let k = k0; k < k1; k++) e += spec[k];
71+ bands.push(Math.log10(e + EPS));
72+ }
73+ const bandMean = bands.reduce((a, v) => a + v, 0) / 4;
74+ for (let b = 0; b < 4; b++) bands[b] -= bandMean;
75+
76+ // Zero-crossing rate over the first 50 ms, in crossings per ms (~kHz).
77+ const zN = Math.min(seg.length, Math.floor(0.05 * sr));
78+ let zc = 0;
79+ for (let i = 1; i < zN; i++) if ((seg[i] >= 0) !== (seg[i - 1] >= 0)) zc++;
80+ const zcr = zN > 1 ? zc / ((zN * 1000) / sr) : 0;
81+
82+ return {
83+ peakDb,
84+ vec: [
85+ Math.log10(centroid + 1), Math.log10(rolloff + 1),
86+ bands[0], bands[1], bands[2], bands[3],
87+ zcr, Math.log10(attackMs + 1), Math.log10(decayMs + 1),
88+ ],
89+ };
90+}
src/analysis/onsets.tsadded+66−0View file
@@ -0,0 +1,66 @@
1+/**
2+ * Onset detection by spectral flux: the half-wave rectified frame-to-frame
3+ * increase in magnitude, summed over bins, peaks where a new sound starts
4+ * and stays flat through sustains and decays. Peaks are picked against a
5+ * local mean plus an absolute floor; the single `sensitivity` control moves
6+ * the floor, since a soft "doo" and a sharp desk tap sit at very different
7+ * flux levels and no fixed threshold serves both.
8+ */
9+import { fft } from '../audio/fft.ts';
10+
11+const FRAME = 1024;
12+const HOP = 256;
13+const MIN_GAP_S = 0.05;
14+
15+/** Onset times in seconds, earliest first. sensitivity in [0, 1]. */
16+export function detectOnsets(x: Float32Array, sr: number, sensitivity: number): number[] {
17+ const nFrames = Math.floor((x.length - FRAME) / HOP);
18+ if (nFrames < 3) return [];
19+
20+ const win = new Float32Array(FRAME);
21+ for (let i = 0; i < FRAME; i++) win[i] = 0.5 - 0.5 * Math.cos((2 * Math.PI * i) / (FRAME - 1));
22+ const re = new Float32Array(FRAME);
23+ const im = new Float32Array(FRAME);
24+ const prev = new Float32Array(FRAME >> 1);
25+ const flux = new Float32Array(nFrames);
26+
27+ for (let f = 0; f < nFrames; f++) {
28+ const off = f * HOP;
29+ for (let i = 0; i < FRAME; i++) { re[i] = x[off + i] * win[i]; im[i] = 0; }
30+ fft(re, im);
31+ let s = 0;
32+ for (let k = 0; k < FRAME >> 1; k++) {
33+ const mag = Math.sqrt(re[k] * re[k] + im[k] * im[k]);
34+ const d = mag - prev[k];
35+ if (d > 0) s += d;
36+ prev[k] = mag;
37+ }
38+ flux[f] = s;
39+ }
40+
41+ let max = 0;
42+ for (let f = 1; f < nFrames; f++) if (flux[f] > max) max = flux[f];
43+ if (max === 0) return [];
44+ for (let f = 0; f < nFrames; f++) flux[f] /= max;
45+ flux[0] = 0; // the first frame's flux is the frame itself, not an onset
46+
47+ // Threshold: local mean over +-W frames, plus a floor the slider moves.
48+ const W = 8;
49+ const floor = 0.02 + 0.28 * (1 - sensitivity);
50+ const minGap = Math.max(1, Math.round((MIN_GAP_S * sr) / HOP));
51+ const onsets: number[] = [];
52+ let last = -minGap;
53+ for (let f = 1; f < nFrames - 1; f++) {
54+ let mean = 0, cnt = 0;
55+ for (let j = Math.max(0, f - W); j <= Math.min(nFrames - 1, f + W); j++) { mean += flux[j]; cnt++; }
56+ mean /= cnt;
57+ if (flux[f] < mean * 1.3 + floor) continue;
58+ if (flux[f] < flux[f - 1] || flux[f] < flux[f + 1]) continue; // local max only
59+ if (f - last < minGap) continue;
60+ last = f;
61+ // The window whose flux peaks started up to a frame before the attack
62+ // it saw; report its centre, which sits on the onset to within a hop.
63+ onsets.push((f * HOP + FRAME / 2) / sr);
64+ }
65+ return onsets;
66+}
src/audio/fft.tsadded+49−0View file
@@ -0,0 +1,49 @@
1+/** In-place radix-2 complex FFT; lengths must be powers of two. Small and
2+ * sufficient for the analysis windows used here (1024 and 4096). */
3+export function fft(re: Float32Array, im: Float32Array): void {
4+ const n = re.length;
5+ if ((n & (n - 1)) !== 0) throw new Error(`fft wants a power of two, got ${n}`);
6+ for (let i = 1, j = 0; i < n; i++) {
7+ let bit = n >> 1;
8+ for (; j & bit; bit >>= 1) j ^= bit;
9+ j ^= bit;
10+ if (i < j) {
11+ let t = re[i]; re[i] = re[j]; re[j] = t;
12+ t = im[i]; im[i] = im[j]; im[j] = t;
13+ }
14+ }
15+ for (let len = 2; len <= n; len <<= 1) {
16+ const half = len >> 1;
17+ const ang = (-2 * Math.PI) / len;
18+ const wr = Math.cos(ang), wi = Math.sin(ang);
19+ for (let i = 0; i < n; i += len) {
20+ let cr = 1, ci = 0;
21+ for (let k = 0; k < half; k++) {
22+ const ar = re[i + k], ai = im[i + k];
23+ const br = re[i + k + half] * cr - im[i + k + half] * ci;
24+ const bi = re[i + k + half] * ci + im[i + k + half] * cr;
25+ re[i + k] = ar + br; im[i + k] = ai + bi;
26+ re[i + k + half] = ar - br; im[i + k + half] = ai - bi;
27+ const t = cr * wr - ci * wi;
28+ ci = cr * wi + ci * wr;
29+ cr = t;
30+ }
31+ }
32+ }
33+}
34+
35+/** Hann-windowed power spectrum of x zero-padded to size n (power of two);
36+ * returns n/2 bins. Bin k is frequency k * sr / n. */
37+export function powerSpectrum(x: Float32Array, n: number): Float32Array {
38+ const re = new Float32Array(n);
39+ const im = new Float32Array(n);
40+ const m = Math.min(x.length, n);
41+ for (let i = 0; i < m; i++) {
42+ const w = 0.5 - 0.5 * Math.cos((2 * Math.PI * i) / (m - 1 || 1));
43+ re[i] = x[i] * w;
44+ }
45+ fft(re, im);
46+ const p = new Float32Array(n >> 1);
47+ for (let k = 0; k < p.length; k++) p[k] = re[k] * re[k] + im[k] * im[k];
48+ return p;
49+}
src/audio/recorder.tsadded+77−0View file
@@ -0,0 +1,77 @@
1+/**
2+ * Microphone capture. The browser's processing chain is asked to stand
3+ * down: automatic gain control would rewrite exactly the relative volumes
4+ * this app exists to track, and noise suppression eats the breathy
5+ * consonants ("tss") that distinguish voices. Capture goes through
6+ * MediaRecorder and is decoded to mono float samples on stop; analysis is
7+ * offline, so codec latency does not matter, and both training and takes
8+ * pass through the identical path.
9+ */
10+export interface Clip {
11+ samples: Float32Array;
12+ sr: number;
13+}
14+
15+export class Recorder {
16+ private stream: MediaStream | null = null;
17+ private rec: MediaRecorder | null = null;
18+ private chunks: Blob[] = [];
19+ private ctx: AudioContext;
20+ private analyser: AnalyserNode | null = null;
21+ private levelBuf = new Float32Array(1024);
22+
23+ constructor(ctx: AudioContext) {
24+ this.ctx = ctx;
25+ }
26+
27+ get recording(): boolean {
28+ return this.rec !== null;
29+ }
30+
31+ async start(): Promise<void> {
32+ this.stream = await navigator.mediaDevices.getUserMedia({
33+ audio: { echoCancellation: false, noiseSuppression: false, autoGainControl: false },
34+ });
35+ if (this.ctx.state === 'suspended') await this.ctx.resume();
36+ const src = this.ctx.createMediaStreamSource(this.stream);
37+ this.analyser = this.ctx.createAnalyser();
38+ this.analyser.fftSize = 1024;
39+ src.connect(this.analyser); // analyser is a sink; nothing reaches the speakers
40+ this.chunks = [];
41+ this.rec = new MediaRecorder(this.stream);
42+ this.rec.ondataavailable = (e) => { if (e.data.size) this.chunks.push(e.data); };
43+ this.rec.start();
44+ }
45+
46+ /** Instantaneous input level in [0, 1], for a meter. */
47+ level(): number {
48+ if (!this.analyser) return 0;
49+ this.analyser.getFloatTimeDomainData(this.levelBuf);
50+ let peak = 0;
51+ for (let i = 0; i < this.levelBuf.length; i++) {
52+ const a = Math.abs(this.levelBuf[i]);
53+ if (a > peak) peak = a;
54+ }
55+ return Math.min(1, peak * 1.4);
56+ }
57+
58+ async stop(): Promise<Clip> {
59+ const rec = this.rec;
60+ if (!rec) throw new Error('not recording');
61+ this.rec = null;
62+ const blob = await new Promise<Blob>((resolve) => {
63+ rec.onstop = () => resolve(new Blob(this.chunks, { type: rec.mimeType }));
64+ rec.stop();
65+ });
66+ this.stream?.getTracks().forEach((t) => t.stop());
67+ this.stream = null;
68+ this.analyser = null;
69+ const buf = await this.ctx.decodeAudioData(await blob.arrayBuffer());
70+ const samples = new Float32Array(buf.length);
71+ for (let c = 0; c < buf.numberOfChannels; c++) {
72+ const ch = buf.getChannelData(c);
73+ for (let i = 0; i < buf.length; i++) samples[i] += ch[i] / buf.numberOfChannels;
74+ }
75+ return { samples, sr: buf.sampleRate };
76+ }
77+}
src/main.tsadded+485−0View file
@@ -0,0 +1,485 @@
1+/**
2+ * The page: voices (named, trained by example), a take (recorded once,
3+ * segmented, classified onto the voices), a timeline (the take as editable
4+ * events), and playback through the renderer seam in render/types.ts. All
5+ * analysis is offline on the recorded clip; nothing here is realtime.
6+ */
7+import type { Clip } from './audio/recorder.ts';
8+import { Recorder } from './audio/recorder.ts';
9+import { detectOnsets } from './analysis/onsets.ts';
10+import { extractHit } from './analysis/features.ts';
11+import { classify, type TrainingSet } from './analysis/classify.ts';
12+import { renderSynth } from './render/synth.ts';
13+import { gainFromPeakDb, type NoteEvent, type SynthPreset } from './render/types.ts';
14+import { WaveView, type Marker } from './ui/waveform.ts';
15+import { Timeline } from './ui/timeline.ts';
16+
17+const PALETTE = ['#e4573d', '#3d7de4', '#3dbb6e', '#d9a520', '#9061d9', '#d95fa8', '#4fb8c9', '#96a13a'];
18+const PRESETS: SynthPreset[] = ['kick', 'snare', 'hat', 'tom'];
19+const STORE_KEY = 'instrawave-compose/v1';
20+
21+interface TrainHit { vec: number[]; peakDb: number; }
22+interface Voice { id: string; name: string; color: string; preset: SynthPreset; train: TrainHit[]; }
23+interface TakeHit {
24+ id: string; t: number; voiceId: string; gain: number;
25+ peakDb: number; vec: number[] | null; // null for hand-added hits
26+ s0: number; s1: number; // sample bounds in the take clip; -1 if hand-added
27+}
28+
29+const $ = <T extends HTMLElement>(id: string) => document.getElementById(id) as T;
30+const uid = () => Math.random().toString(36).slice(2, 10);
31+
32+// ---- state ----------------------------------------------------------------
33+
34+let voices: Voice[] = loadVoices();
35+let take: { clip: Clip | null; hits: TakeHit[] } = { clip: null, hits: [] };
36+let pending: { clip: Clip; voiceId: string; onsets: number[]; excluded: Set<number> } | null = null;
37+let recTarget: { kind: 'voice'; voiceId: string } | { kind: 'take' } | null = null;
38+
39+let actx: AudioContext | null = null;
40+const audio = (): AudioContext => (actx ??= new AudioContext());
41+let recorder: Recorder | null = null;
42+let playing: { src: AudioBufferSourceNode; t0: number; dur: number } | null = null;
43+
44+function loadVoices(): Voice[] {
45+ try {
46+ const raw = localStorage.getItem(STORE_KEY);
47+ if (raw) return (JSON.parse(raw) as { voices: Voice[] }).voices;
48+ } catch { /* a corrupt store starts fresh */ }
49+ return [];
50+}
51+function save(): void {
52+ localStorage.setItem(STORE_KEY, JSON.stringify({ voices }));
53+}
54+function voiceById(id: string): Voice | undefined {
55+ return voices.find((v) => v.id === id);
56+}
57+function status(msg: string): void {
58+ $('status').textContent = msg;
59+}
60+
61+// ---- audition -------------------------------------------------------------
62+
63+function playSlice(clip: Clip, s0: number, s1: number): void {
64+ const ctx = audio();
65+ const n = Math.min(s1, s0 + Math.floor(0.5 * clip.sr)) - s0;
66+ if (n <= 0) return;
67+ const buf = ctx.createBuffer(1, n, clip.sr);
68+ buf.getChannelData(0).set(clip.samples.subarray(s0, s0 + n));
69+ const src = ctx.createBufferSource();
70+ src.buffer = buf;
71+ src.connect(ctx.destination);
72+ src.start();
73+}
74+
75+async function playBuffer(buf: AudioBuffer, onFrame?: (t: number | null) => void): Promise<void> {
76+ const ctx = audio();
77+ if (ctx.state === 'suspended') await ctx.resume();
78+ const src = ctx.createBufferSource();
79+ src.buffer = buf;
80+ src.connect(ctx.destination);
81+ const t0 = ctx.currentTime;
82+ playing = { src, t0, dur: buf.duration };
83+ src.onended = () => { if (playing?.src === src) playing = null; onFrame?.(null); syncPlayButton(); };
84+ src.start();
85+ if (onFrame) {
86+ const tick = () => {
87+ if (!playing || playing.src !== src) return;
88+ onFrame(ctx.currentTime - t0);
89+ requestAnimationFrame(tick);
90+ };
91+ requestAnimationFrame(tick);
92+ }
93+ syncPlayButton();
94+}
95+
96+function stopPlayback(): void {
97+ playing?.src.stop();
98+ playing = null;
99+ timeline.playhead(null);
100+ syncPlayButton();
101+}
102+
103+// ---- recording ------------------------------------------------------------
104+
105+let meterRaf = 0;
106+function meterOn(): void {
107+ $('meter').hidden = false;
108+ const fill = $('meter').firstElementChild as HTMLElement;
109+ const tick = () => {
110+ fill.style.width = `${(recorder?.level() ?? 0) * 100}%`;
111+ if (recorder?.recording) meterRaf = requestAnimationFrame(tick);
112+ };
113+ meterRaf = requestAnimationFrame(tick);
114+}
115+function meterOff(): void {
116+ cancelAnimationFrame(meterRaf);
117+ $('meter').hidden = true;
118+}
119+
120+async function startRecording(target: NonNullable<typeof recTarget>): Promise<void> {
121+ recorder = new Recorder(audio());
122+ try {
123+ await recorder.start();
124+ } catch (e) {
125+ status(`microphone unavailable: ${e instanceof Error ? e.message : e}`);
126+ recorder = null;
127+ return;
128+ }
129+ recTarget = target;
130+ meterOn();
131+ renderVoiceList();
132+ syncTakeBar();
133+ status(target.kind === 'take'
134+ ? 'recording the take; stop when the phrase is done'
135+ : `recording examples for "${voiceById(target.voiceId)?.name}"; make the sound several times, then stop`);
136+}
137+
138+async function stopRecording(): Promise<void> {
139+ if (!recorder) return;
140+ const target = recTarget!;
141+ let clip: Clip;
142+ try {
143+ clip = await recorder.stop();
144+ } finally {
145+ recorder = null;
146+ recTarget = null;
147+ meterOff();
148+ }
149+ renderVoiceList();
150+ syncTakeBar();
151+ if (target.kind === 'voice') {
152+ pending = { clip, voiceId: target.voiceId, onsets: [], excluded: new Set() };
153+ redetectPending();
154+ $('capture').hidden = false;
155+ } else {
156+ take.clip = clip;
157+ processTake();
158+ }
159+}
160+
161+// ---- capture review (voice training) ---------------------------------------
162+
163+const waveView = new WaveView($('capture-wave') as HTMLCanvasElement);
164+
165+function redetectPending(): void {
166+ if (!pending) return;
167+ const sens = Number(($('capture-sens') as HTMLInputElement).value);
168+ pending.onsets = detectOnsets(pending.clip.samples, pending.clip.sr, sens);
169+ pending.excluded = new Set();
170+ refreshCapture();
171+}
172+
173+function refreshCapture(): void {
174+ if (!pending) return;
175+ const voice = voiceById(pending.voiceId)!;
176+ const markers: Marker[] = pending.onsets.map((t, i) => ({ t, excluded: pending!.excluded.has(i) }));
177+ waveView.set(pending.clip, markers, voice.color);
178+ const n = pending.onsets.length - pending.excluded.size;
179+ $('capture-title').textContent = `${pending.onsets.length} hit${pending.onsets.length === 1 ? '' : 's'} detected`;
180+ ($('capture-commit') as HTMLButtonElement).textContent = `Add ${n} hit${n === 1 ? '' : 's'} to "${voice.name}"`;
181+ ($('capture-commit') as HTMLButtonElement).disabled = n === 0;
182+ ($('capture-exclude') as HTMLButtonElement).disabled = true;
183+}
184+
185+waveView.onMarkerClick = (i) => {
186+ if (!pending) return;
187+ waveView.selected = i;
188+ waveView.draw();
189+ const s0 = Math.floor(pending.onsets[i] * pending.clip.sr);
190+ const s1 = i + 1 < pending.onsets.length ? Math.floor(pending.onsets[i + 1] * pending.clip.sr) : pending.clip.samples.length;
191+ playSlice(pending.clip, s0, s1);
192+ const btn = $('capture-exclude') as HTMLButtonElement;
193+ btn.disabled = false;
194+ btn.textContent = pending.excluded.has(i) ? 'Include' : 'Exclude';
195+};
196+
197+$('capture-exclude').addEventListener('click', () => {
198+ const i = waveView.selected;
199+ if (!pending || i < 0) return;
200+ if (pending.excluded.has(i)) pending.excluded.delete(i);
201+ else pending.excluded.add(i);
202+ const sel = i;
203+ refreshCapture();
204+ waveView.selected = sel;
205+ waveView.draw();
206+ const btn = $('capture-exclude') as HTMLButtonElement;
207+ btn.disabled = false;
208+ btn.textContent = pending.excluded.has(i) ? 'Include' : 'Exclude';
209+});
210+
211+$('capture-sens').addEventListener('input', redetectPending);
212+
213+$('capture-commit').addEventListener('click', () => {
214+ if (!pending) return;
215+ const voice = voiceById(pending.voiceId)!;
216+ const { samples, sr } = pending.clip;
217+ let added = 0;
218+ for (let i = 0; i < pending.onsets.length; i++) {
219+ if (pending.excluded.has(i)) continue;
220+ const s0 = Math.floor(pending.onsets[i] * sr);
221+ const s1 = i + 1 < pending.onsets.length ? Math.floor(pending.onsets[i + 1] * sr) : samples.length;
222+ const f = extractHit(samples, sr, s0, s1);
223+ voice.train.push({ vec: f.vec, peakDb: f.peakDb });
224+ added++;
225+ }
226+ save();
227+ pending = null;
228+ $('capture').hidden = true;
229+ renderVoiceList();
230+ status(`added ${added} hit${added === 1 ? '' : 's'} to "${voice.name}" (${voice.train.length} total)`);
231+});
232+
233+$('capture-discard').addEventListener('click', () => {
234+ pending = null;
235+ $('capture').hidden = true;
236+});
237+
238+// ---- voices ---------------------------------------------------------------
239+
240+function renderVoiceList(): void {
241+ const list = $('voice-list');
242+ list.textContent = '';
243+ const busy = recorder !== null;
244+ for (const voice of voices) {
245+ const row = document.createElement('div');
246+ row.className = 'voice-row';
247+
248+ const swatch = document.createElement('span');
249+ swatch.className = 'swatch';
250+ swatch.style.background = voice.color;
251+
252+ const name = document.createElement('input');
253+ name.className = 'name';
254+ name.value = voice.name;
255+ name.addEventListener('change', () => {
256+ voice.name = name.value.trim() || voice.name;
257+ name.value = voice.name;
258+ save();
259+ refreshTimeline();
260+ });
261+
262+ const count = document.createElement('span');
263+ count.className = 'count';
264+ count.textContent = `${voice.train.length} trained hit${voice.train.length === 1 ? '' : 's'}`;
265+
266+ const preset = document.createElement('select');
267+ for (const p of PRESETS) {
268+ const o = document.createElement('option');
269+ o.value = p; o.textContent = p;
270+ preset.appendChild(o);
271+ }
272+ preset.value = voice.preset;
273+ preset.addEventListener('change', () => { voice.preset = preset.value as SynthPreset; save(); });
274+
275+ const rec = document.createElement('button');
276+ const recordingThis = recTarget?.kind === 'voice' && recTarget.voiceId === voice.id;
277+ rec.innerHTML = recordingThis ? '■ Stop' : '● Record';
278+ if (recordingThis) rec.className = 'rec';
279+ rec.disabled = busy && !recordingThis;
280+ rec.addEventListener('click', () => {
281+ if (recordingThis) void stopRecording();
282+ else void startRecording({ kind: 'voice', voiceId: voice.id });
283+ });
284+
285+ const clear = document.createElement('button');
286+ clear.textContent = 'Clear';
287+ clear.title = 'forget this voice\'s trained hits';
288+ clear.disabled = voice.train.length === 0;
289+ clear.addEventListener('click', () => {
290+ voice.train = [];
291+ save();
292+ renderVoiceList();
293+ });
294+
295+ const del = document.createElement('button');
296+ del.textContent = '✕';
297+ del.title = 'delete voice';
298+ del.addEventListener('click', () => {
299+ if (!confirm(`Delete voice "${voice.name}"? Its trained hits and its timeline events go with it.`)) return;
300+ voices = voices.filter((v) => v.id !== voice.id);
301+ take.hits = take.hits.filter((h) => h.voiceId !== voice.id);
302+ save();
303+ renderVoiceList();
304+ refreshTimeline();
305+ });
306+
307+ const spacer = document.createElement('span');
308+ spacer.className = 'spacer';
309+ row.append(swatch, name, count, rec, clear, spacer, preset, del);
310+ list.appendChild(row);
311+ }
312+}
313+
314+$('add-voice').addEventListener('click', () => {
315+ const input = $('voice-name') as HTMLInputElement;
316+ const name = input.value.trim() || `voice ${voices.length + 1}`;
317+ voices.push({
318+ id: uid(),
319+ name,
320+ color: PALETTE[voices.length % PALETTE.length],
321+ preset: PRESETS[voices.length % PRESETS.length],
322+ train: [],
323+ });
324+ input.value = '';
325+ save();
326+ renderVoiceList();
327+ refreshTimeline();
328+});
329+
330+$('reset-all').addEventListener('click', () => {
331+ if (!confirm('Forget all voices, training, and the current take?')) return;
332+ localStorage.removeItem(STORE_KEY);
333+ voices = [];
334+ take = { clip: null, hits: [] };
335+ pending = null;
336+ $('capture').hidden = true;
337+ renderVoiceList();
338+ refreshTimeline();
339+ syncTakeBar();
340+ status('');
341+});
342+
343+// ---- take -----------------------------------------------------------------
344+
345+const timeline = new Timeline($('timeline') as HTMLCanvasElement);
346+
347+function processTake(): void {
348+ if (!take.clip) return;
349+ stopPlayback();
350+ const { samples, sr } = take.clip;
351+ const sens = Number(($('take-sens') as HTMLInputElement).value);
352+ const onsets = detectOnsets(samples, sr, sens);
353+ const trained = voices.filter((v) => v.train.length > 0);
354+ const trainSet: TrainingSet = {
355+ voiceIds: trained.flatMap((v) => v.train.map(() => v.id)),
356+ vecs: trained.flatMap((v) => v.train.map((h) => h.vec)),
357+ };
358+
359+ const hits: TakeHit[] = [];
360+ let refDb = -160;
361+ for (let i = 0; i < onsets.length; i++) {
362+ const s0 = Math.floor(onsets[i] * sr);
363+ const s1 = i + 1 < onsets.length ? Math.floor(onsets[i + 1] * sr) : samples.length;
364+ const f = extractHit(samples, sr, s0, s1);
365+ if (f.peakDb > refDb) refDb = f.peakDb;
366+ const cls = classify(f.vec, trainSet);
367+ const voiceId = cls?.voiceId ?? voices[0]?.id;
368+ if (!voiceId) continue;
369+ hits.push({ id: uid(), t: onsets[i], voiceId, gain: 0, peakDb: f.peakDb, vec: f.vec, s0, s1 });
370+ }
371+ for (const h of hits) h.gain = gainFromPeakDb(h.peakDb, refDb);
372+ take.hits = hits;
373+ refreshTimeline();
374+ syncTakeBar();
375+ if (voices.length === 0) status('no voices defined; add and train a voice, then hit re-detect');
376+ else if (trainSet.vecs.length === 0) status(`no trained hits yet; all ${hits.length} events landed on "${voices[0].name}"`);
377+ else status(`${hits.length} hits classified across ${trained.length} trained voice${trained.length === 1 ? '' : 's'}`);
378+}
379+
380+function refreshTimeline(): void {
381+ timeline.setData(
382+ voices.map((v) => ({ id: v.id, name: v.name, color: v.color })),
383+ take.hits,
384+ take.clip,
385+ );
386+ syncHitButtons();
387+}
388+
389+function syncTakeBar(): void {
390+ const recordingTake = recTarget?.kind === 'take';
391+ const btn = $('take-record') as HTMLButtonElement;
392+ btn.innerHTML = recordingTake ? '■ Stop' : '● Record take';
393+ btn.className = recordingTake ? 'rec' : '';
394+ btn.disabled = recorder !== null && !recordingTake;
395+ ($('take-redetect') as HTMLButtonElement).disabled = take.clip === null;
396+ ($('take-play') as HTMLButtonElement).disabled = take.hits.length === 0;
397+}
398+
399+function syncPlayButton(): void {
400+ ($('take-play') as HTMLButtonElement).textContent = playing ? 'Stop' : 'Play';
401+ ($('take-play') as HTMLButtonElement).disabled = take.hits.length === 0 && !playing;
402+}
403+
404+function syncHitButtons(): void {
405+ const sel = take.hits.find((h) => h.id === timeline.selected);
406+ ($('hit-delete') as HTMLButtonElement).disabled = !sel;
407+ ($('hit-learn') as HTMLButtonElement).disabled = !sel || sel.vec === null;
408+}
409+
410+$('take-record').addEventListener('click', () => {
411+ if (recTarget?.kind === 'take') void stopRecording();
412+ else void startRecording({ kind: 'take' });
413+});
414+
415+$('take-redetect').addEventListener('click', processTake);
416+$('take-sens').addEventListener('change', () => { if (take.clip) processTake(); });
417+
418+timeline.onSelect = (id) => {
419+ syncHitButtons();
420+ const hit = take.hits.find((h) => h.id === id);
421+ if (!hit) return;
422+ if (hit.s0 >= 0 && take.clip) playSlice(take.clip, hit.s0, hit.s1);
423+ else void auditionSynth(hit);
424+};
425+
426+async function auditionSynth(hit: TakeHit): Promise<void> {
427+ const voice = voiceById(hit.voiceId);
428+ if (!voice) return;
429+ const buf = await renderSynth([{ t: 0, voiceId: hit.voiceId, gain: hit.gain }], () => voice.preset, 1);
430+ void playBuffer(buf);
431+}
432+
433+timeline.onReassign = (id, voiceId) => {
434+ const voice = voiceById(voiceId);
435+ status(voice ? `hit moved to "${voice.name}"; Learn adds it to that voice's training` : '');
436+ syncHitButtons();
437+};
438+
439+timeline.onAdd = (t, voiceId) => {
440+ take.hits.push({ id: uid(), t, voiceId, gain: 0.8, peakDb: 0, vec: null, s0: -1, s1: -1 });
441+ take.hits.sort((a, b) => a.t - b.t);
442+ refreshTimeline();
443+};
444+
445+$('hit-delete').addEventListener('click', deleteSelected);
446+document.addEventListener('keydown', (e) => {
447+ if (e.key !== 'Delete' && e.key !== 'Backspace') return;
448+ if (document.activeElement && ['INPUT', 'SELECT', 'TEXTAREA'].includes(document.activeElement.tagName)) return;
449+ deleteSelected();
450+});
451+function deleteSelected(): void {
452+ if (!timeline.selected) return;
453+ take.hits = take.hits.filter((h) => h.id !== timeline.selected);
454+ timeline.selected = null;
455+ refreshTimeline();
456+}
457+
458+$('hit-learn').addEventListener('click', () => {
459+ const hit = take.hits.find((h) => h.id === timeline.selected);
460+ if (!hit || !hit.vec) return;
461+ const voice = voiceById(hit.voiceId);
462+ if (!voice) return;
463+ voice.train.push({ vec: hit.vec, peakDb: hit.peakDb });
464+ save();
465+ renderVoiceList();
466+ status(`learned this hit as "${voice.name}" (${voice.train.length} trained)`);
467+});
468+
469+$('take-play').addEventListener('click', async () => {
470+ if (playing) { stopPlayback(); return; }
471+ if (take.hits.length === 0) return;
472+ const events: NoteEvent[] = take.hits.map((h) => ({ t: h.t, voiceId: h.voiceId, gain: h.gain }));
473+ const seconds = Math.max(...events.map((e) => e.t)) + 1.5;
474+ const buf = await renderSynth(events, (id) => voiceById(id)?.preset ?? 'kick', seconds);
475+ void playBuffer(buf, (t) => timeline.playhead(t));
476+});
477+
478+// ---- boot -----------------------------------------------------------------
479+
480+renderVoiceList();
481+refreshTimeline();
482+syncTakeBar();
483+if (voices.length === 0) {
484+ status('start by adding a voice, then record yourself making its sound a handful of times');
485+}
src/render/synth.tsadded+92−0View file
@@ -0,0 +1,92 @@
1+/**
2+ * Placeholder drum voices on an OfflineAudioContext: a pitch-dropping sine
3+ * for the kick and tom, filtered noise for the snare and hat. These exist
4+ * so the capture-classify-arrange loop is playable end to end; the
5+ * instrawave renderer replaces them behind the same interface.
6+ */
7+import type { NoteEvent, SynthPreset } from './types.ts';
8+
9+const SR = 44100;
10+
11+export async function renderSynth(
12+ events: NoteEvent[],
13+ preset: (voiceId: string) => SynthPreset,
14+ seconds: number,
15+): Promise<AudioBuffer> {
16+ const len = Math.max(1, Math.ceil(seconds * SR));
17+ const ctx = new OfflineAudioContext(1, len, SR);
18+ const noise = noiseBuffer(ctx);
19+ for (const ev of events) {
20+ const g = Math.max(1e-4, Math.min(1, ev.gain));
21+ switch (preset(ev.voiceId)) {
22+ case 'kick': kick(ctx, ev.t, g); break;
23+ case 'snare': snare(ctx, ev.t, g, noise); break;
24+ case 'hat': hat(ctx, ev.t, g, noise); break;
25+ case 'tom': tom(ctx, ev.t, g); break;
26+ }
27+ }
28+ return ctx.startRendering();
29+}
30+
31+function noiseBuffer(ctx: OfflineAudioContext): AudioBuffer {
32+ const buf = ctx.createBuffer(1, SR, SR);
33+ const x = buf.getChannelData(0);
34+ let s = 22222; // deterministic; renders are reproducible
35+ for (let i = 0; i < x.length; i++) {
36+ s = (s * 1103515245 + 12345) & 0x7fffffff;
37+ x[i] = (s / 0x3fffffff) - 1;
38+ }
39+ return buf;
40+}
41+
42+function env(ctx: OfflineAudioContext, t: number, g: number, decay: number): GainNode {
43+ const gain = ctx.createGain();
44+ gain.gain.setValueAtTime(g, t);
45+ gain.gain.exponentialRampToValueAtTime(1e-4, t + decay);
46+ gain.connect(ctx.destination);
47+ return gain;
48+}
49+
50+function kick(ctx: OfflineAudioContext, t: number, g: number): void {
51+ const osc = ctx.createOscillator();
52+ osc.frequency.setValueAtTime(150, t);
53+ osc.frequency.exponentialRampToValueAtTime(46, t + 0.09);
54+ osc.connect(env(ctx, t, g, 0.45));
55+ osc.start(t);
56+ osc.stop(t + 0.5);
57+}
58+
59+function tom(ctx: OfflineAudioContext, t: number, g: number): void {
60+ const osc = ctx.createOscillator();
61+ osc.frequency.setValueAtTime(150, t);
62+ osc.frequency.exponentialRampToValueAtTime(98, t + 0.12);
63+ osc.connect(env(ctx, t, g * 0.9, 0.3));
64+ osc.start(t);
65+ osc.stop(t + 0.35);
66+}
67+
68+function snare(ctx: OfflineAudioContext, t: number, g: number, noise: AudioBuffer): void {
69+ const tone = ctx.createOscillator();
70+ tone.type = 'triangle';
71+ tone.frequency.value = 185;
72+ tone.connect(env(ctx, t, g * 0.4, 0.12));
73+ tone.start(t);
74+ tone.stop(t + 0.15);
75+ const n = ctx.createBufferSource();
76+ n.buffer = noise;
77+ const bp = ctx.createBiquadFilter();
78+ bp.type = 'highpass';
79+ bp.frequency.value = 900;
80+ n.connect(bp).connect(env(ctx, t, g * 0.7, 0.2));
81+ n.start(t, 0, 0.25);
82+}
83+
84+function hat(ctx: OfflineAudioContext, t: number, g: number, noise: AudioBuffer): void {
85+ const n = ctx.createBufferSource();
86+ n.buffer = noise;
87+ const hp = ctx.createBiquadFilter();
88+ hp.type = 'highpass';
89+ hp.frequency.value = 7000;
90+ n.connect(hp).connect(env(ctx, t, g * 0.5, 0.06));
91+ n.start(t, 0, 0.1);
92+}
src/render/types.tsadded+27−0View file
@@ -0,0 +1,27 @@
1+/**
2+ * The seam between the analysis half and the sound half. Everything above
3+ * this interface deals in NoteEvents (a time, a voice, a strength);
4+ * everything below turns them into audio. The synth renderer is the first
5+ * implementation; the instrawave renderer replaces a binding's preset with
6+ * an instrument file plus parameter values, groups events by bake key so
7+ * voices sharing a bake render in one pass, and mixes the traces.
8+ */
9+export interface NoteEvent {
10+ t: number; // seconds from the start of the take
11+ voiceId: string;
12+ gain: number; // linear, 0..1, from the hit's measured loudness
13+}
14+
15+export type SynthPreset = 'kick' | 'snare' | 'hat' | 'tom';
16+
17+export interface Renderer {
18+ render(events: NoteEvent[], preset: (voiceId: string) => SynthPreset, seconds: number): Promise<AudioBuffer>;
19+}
20+
21+/** Map a hit's peak level to a note strength. Levels are taken relative to
22+ * the loudest hit in the take and compressed (0.5 exponent over a 30 dB
23+ * window), so soft hits stay audible while the ordering is preserved. */
24+export function gainFromPeakDb(peakDb: number, refDb: number): number {
25+ const rel = Math.max(-30, Math.min(0, peakDb - refDb));
26+ return Math.pow(10 ** (rel / 20), 0.5);
27+}
src/ui/timeline.tsadded+217−0View file
@@ -0,0 +1,217 @@
1+/**
2+ * The take as an editable timeline: the recording's waveform in a strip at
3+ * the top, one lane per voice below, a marker per hit whose height is its
4+ * strength. Times are exactly as performed; the only edits are the ones a
5+ * wrong classification needs. Click selects and auditions; dragging a
6+ * marker to another lane reassigns it; double-clicking an empty spot in a
7+ * lane adds a hit there.
8+ */
9+import type { Clip } from '../audio/recorder.ts';
10+
11+export interface Lane { id: string; name: string; color: string; }
12+export interface TLEvent { id: string; t: number; voiceId: string; gain: number; }
13+
14+const BG = '#16181d';
15+const WAVE = '#44546a';
16+const STRIP_H = 44;
17+const LANE_H = 42;
18+const AXIS_H = 18;
19+const GUTTER = 92;
20+
21+export class Timeline {
22+ private canvas: HTMLCanvasElement;
23+ private lanes: Lane[] = [];
24+ private events: TLEvent[] = [];
25+ private clip: Clip | null = null;
26+ private duration = 1;
27+ private playT: number | null = null;
28+ private drag: { id: string; moved: boolean } | null = null;
29+ selected: string | null = null;
30+ onSelect: ((id: string | null) => void) | null = null;
31+ onReassign: ((id: string, voiceId: string) => void) | null = null;
32+ onAdd: ((t: number, voiceId: string) => void) | null = null;
33+
34+ constructor(canvas: HTMLCanvasElement) {
35+ this.canvas = canvas;
36+ canvas.addEventListener('pointerdown', (e) => this.pointerDown(e));
37+ canvas.addEventListener('pointermove', (e) => this.pointerMove(e));
38+ canvas.addEventListener('pointerup', (e) => this.pointerUp(e));
39+ canvas.addEventListener('dblclick', (e) => this.dblClick(e));
40+ new ResizeObserver(() => this.draw()).observe(canvas);
41+ }
42+
43+ setData(lanes: Lane[], events: TLEvent[], clip: Clip | null): void {
44+ this.lanes = lanes;
45+ this.events = events;
46+ this.clip = clip;
47+ const lastT = events.reduce((a, e) => Math.max(a, e.t), 0);
48+ this.duration = Math.max(clip ? clip.samples.length / clip.sr : 0, lastT + 0.5, 1);
49+ this.canvas.style.height = `${STRIP_H + lanes.length * LANE_H + AXIS_H}px`;
50+ if (this.selected && !events.some((e) => e.id === this.selected)) this.selected = null;
51+ this.draw();
52+ }
53+
54+ playhead(t: number | null): void {
55+ this.playT = t;
56+ this.draw();
57+ }
58+
59+ private xOf(t: number, w: number): number {
60+ return GUTTER + (t / this.duration) * (w - GUTTER - 8);
61+ }
62+ private tOf(x: number, w: number): number {
63+ return ((x - GUTTER) / (w - GUTTER - 8)) * this.duration;
64+ }
65+ private laneOf(y: number): number {
66+ return Math.floor((y - STRIP_H) / LANE_H);
67+ }
68+
69+ private hitAt(x: number, y: number): TLEvent | null {
70+ const w = this.canvas.clientWidth;
71+ const li = this.laneOf(y);
72+ let best: TLEvent | null = null, bestDx = 7;
73+ for (const ev of this.events) {
74+ if (this.lanes.findIndex((l) => l.id === ev.voiceId) !== li) continue;
75+ const dx = Math.abs(this.xOf(ev.t, w) - x);
76+ if (dx < bestDx) { best = ev; bestDx = dx; }
77+ }
78+ return best;
79+ }
80+
81+ private pointerDown(e: PointerEvent): void {
82+ const rect = this.canvas.getBoundingClientRect();
83+ const ev = this.hitAt(e.clientX - rect.left, e.clientY - rect.top);
84+ if (ev) {
85+ this.drag = { id: ev.id, moved: false };
86+ this.canvas.setPointerCapture(e.pointerId);
87+ if (this.selected !== ev.id) {
88+ this.selected = ev.id;
89+ this.onSelect?.(ev.id);
90+ } else this.onSelect?.(ev.id); // re-click re-auditions
91+ } else {
92+ this.selected = null;
93+ this.onSelect?.(null);
94+ }
95+ this.draw();
96+ }
97+
98+ private pointerMove(e: PointerEvent): void {
99+ if (!this.drag) return;
100+ const rect = this.canvas.getBoundingClientRect();
101+ const li = Math.max(0, Math.min(this.lanes.length - 1, this.laneOf(e.clientY - rect.top)));
102+ const ev = this.events.find((v) => v.id === this.drag!.id);
103+ if (ev && this.lanes[li] && ev.voiceId !== this.lanes[li].id) {
104+ ev.voiceId = this.lanes[li].id;
105+ this.drag.moved = true;
106+ this.draw();
107+ }
108+ }
109+
110+ private pointerUp(e: PointerEvent): void {
111+ if (this.drag?.moved) {
112+ const ev = this.events.find((v) => v.id === this.drag!.id);
113+ if (ev) this.onReassign?.(ev.id, ev.voiceId);
114+ }
115+ this.drag = null;
116+ if (this.canvas.hasPointerCapture(e.pointerId)) this.canvas.releasePointerCapture(e.pointerId);
117+ }
118+
119+ private dblClick(e: PointerEvent | MouseEvent): void {
120+ const rect = this.canvas.getBoundingClientRect();
121+ const x = e.clientX - rect.left, y = e.clientY - rect.top;
122+ if (this.hitAt(x, y)) return;
123+ const li = this.laneOf(y);
124+ if (li < 0 || li >= this.lanes.length || x < GUTTER) return;
125+ const t = Math.max(0, this.tOf(x, this.canvas.clientWidth));
126+ this.onAdd?.(t, this.lanes[li].id);
127+ }
128+
129+ draw(): void {
130+ const c = this.canvas;
131+ const dpr = devicePixelRatio || 1;
132+ const w = c.clientWidth, h = c.clientHeight;
133+ if (w === 0 || h === 0) return;
134+ if (c.width !== Math.round(w * dpr) || c.height !== Math.round(h * dpr)) {
135+ c.width = Math.round(w * dpr);
136+ c.height = Math.round(h * dpr);
137+ }
138+ const g = c.getContext('2d')!;
139+ g.setTransform(dpr, 0, 0, dpr, 0, 0);
140+ g.fillStyle = BG;
141+ g.fillRect(0, 0, w, h);
142+
143+ // Waveform strip.
144+ if (this.clip) {
145+ const { samples, sr } = this.clip;
146+ const mid = STRIP_H / 2;
147+ g.strokeStyle = WAVE;
148+ g.lineWidth = 1;
149+ g.beginPath();
150+ for (let px = GUTTER; px < w - 8; px++) {
151+ const t0 = this.tOf(px, w), t1 = this.tOf(px + 1, w);
152+ const i0 = Math.max(0, Math.floor(t0 * sr)), i1 = Math.min(samples.length, Math.floor(t1 * sr));
153+ let lo = 0, hi = 0;
154+ for (let i = i0; i < i1; i++) {
155+ if (samples[i] < lo) lo = samples[i];
156+ if (samples[i] > hi) hi = samples[i];
157+ }
158+ g.moveTo(px + 0.5, mid - hi * mid * 0.9 - 0.5);
159+ g.lineTo(px + 0.5, mid - lo * mid * 0.9 + 0.5);
160+ }
161+ g.stroke();
162+ }
163+
164+ // Lanes.
165+ for (let i = 0; i < this.lanes.length; i++) {
166+ const y0 = STRIP_H + i * LANE_H;
167+ g.fillStyle = i % 2 ? 'rgba(255,255,255,0.028)' : 'rgba(255,255,255,0.055)';
168+ g.fillRect(0, y0, w, LANE_H);
169+ g.fillStyle = this.lanes[i].color;
170+ g.fillRect(0, y0, 3, LANE_H);
171+ g.font = '12px system-ui, sans-serif';
172+ g.fillStyle = 'rgba(255,255,255,0.85)';
173+ g.fillText(this.lanes[i].name.slice(0, 12), 10, y0 + LANE_H / 2 + 4);
174+ }
175+
176+ // Time axis.
177+ const axisY = STRIP_H + this.lanes.length * LANE_H;
178+ g.font = '10px system-ui, sans-serif';
179+ g.fillStyle = 'rgba(255,255,255,0.5)';
180+ const step = this.duration > 8 ? 2 : this.duration > 3 ? 1 : 0.5;
181+ for (let t = 0; t <= this.duration; t += step) {
182+ const x = this.xOf(t, w);
183+ g.fillRect(x, axisY, 1, 4);
184+ g.fillText(`${t.toFixed(step < 1 ? 1 : 0)}s`, x + 2, axisY + 13);
185+ }
186+
187+ // Events.
188+ for (const ev of this.events) {
189+ const li = this.lanes.findIndex((l) => l.id === ev.voiceId);
190+ if (li < 0) continue;
191+ const x = this.xOf(ev.t, w);
192+ const y0 = STRIP_H + li * LANE_H;
193+ const hBar = 6 + ev.gain * (LANE_H - 12);
194+ const sel = ev.id === this.selected;
195+ g.fillStyle = this.lanes[li].color;
196+ g.globalAlpha = sel ? 1 : 0.85;
197+ g.fillRect(x - (sel ? 2.5 : 1.5), y0 + LANE_H - 3 - hBar, sel ? 5 : 3, hBar);
198+ if (sel) {
199+ g.strokeStyle = '#fff';
200+ g.lineWidth = 1;
201+ g.strokeRect(x - 4.5, y0 + 2.5, 9, LANE_H - 5);
202+ }
203+ g.globalAlpha = 1;
204+ }
205+
206+ // Playhead.
207+ if (this.playT !== null) {
208+ const x = this.xOf(this.playT, w);
209+ g.strokeStyle = 'rgba(255,255,255,0.8)';
210+ g.lineWidth = 1;
211+ g.beginPath();
212+ g.moveTo(x, 0);
213+ g.lineTo(x, axisY);
214+ g.stroke();
215+ }
216+ }
217+}
src/ui/waveform.tsadded+103−0View file
@@ -0,0 +1,103 @@
1+/**
2+ * The capture review: one recorded clip drawn as a waveform with a marker
3+ * per detected onset. Markers audition on click and can be excluded (drawn
4+ * hollow) before the hits are committed to a voice's training set, so a
5+ * cough or a spurious detection is visible and removable rather than
6+ * silently poisoning the label.
7+ */
8+import type { Clip } from '../audio/recorder.ts';
9+
10+export interface Marker {
11+ t: number;
12+ excluded: boolean;
13+}
14+
15+const BG = '#16181d';
16+const WAVE = '#5b8dbf';
17+
18+export class WaveView {
19+ private canvas: HTMLCanvasElement;
20+ private clip: Clip | null = null;
21+ private markers: Marker[] = [];
22+ private color = '#e4573d';
23+ selected = -1;
24+ onMarkerClick: ((index: number) => void) | null = null;
25+
26+ constructor(canvas: HTMLCanvasElement) {
27+ this.canvas = canvas;
28+ canvas.addEventListener('pointerdown', (e) => {
29+ if (!this.clip) return;
30+ const rect = canvas.getBoundingClientRect();
31+ const x = e.clientX - rect.left;
32+ const dur = this.clip.samples.length / this.clip.sr;
33+ let best = -1, bestDx = 8;
34+ for (let i = 0; i < this.markers.length; i++) {
35+ const mx = (this.markers[i].t / dur) * rect.width;
36+ const dx = Math.abs(mx - x);
37+ if (dx < bestDx) { best = i; bestDx = dx; }
38+ }
39+ if (best >= 0) this.onMarkerClick?.(best);
40+ });
41+ new ResizeObserver(() => this.draw()).observe(canvas);
42+ }
43+
44+ set(clip: Clip | null, markers: Marker[], color: string): void {
45+ this.clip = clip;
46+ this.markers = markers;
47+ this.color = color;
48+ this.selected = -1;
49+ this.draw();
50+ }
51+
52+ draw(): void {
53+ const c = this.canvas;
54+ const dpr = devicePixelRatio || 1;
55+ const w = c.clientWidth, h = c.clientHeight;
56+ if (w === 0) return;
57+ if (c.width !== Math.round(w * dpr) || c.height !== Math.round(h * dpr)) {
58+ c.width = Math.round(w * dpr);
59+ c.height = Math.round(h * dpr);
60+ }
61+ const g = c.getContext('2d')!;
62+ g.setTransform(dpr, 0, 0, dpr, 0, 0);
63+ g.fillStyle = BG;
64+ g.fillRect(0, 0, w, h);
65+ if (!this.clip) return;
66+
67+ const { samples, sr } = this.clip;
68+ const dur = samples.length / sr;
69+ const mid = h / 2;
70+ g.strokeStyle = WAVE;
71+ g.lineWidth = 1;
72+ g.beginPath();
73+ for (let px = 0; px < w; px++) {
74+ const i0 = Math.floor((px / w) * samples.length);
75+ const i1 = Math.floor(((px + 1) / w) * samples.length);
76+ let lo = 0, hi = 0;
77+ for (let i = i0; i < i1; i++) {
78+ if (samples[i] < lo) lo = samples[i];
79+ if (samples[i] > hi) hi = samples[i];
80+ }
81+ g.moveTo(px + 0.5, mid - hi * mid * 0.92 - 0.5);
82+ g.lineTo(px + 0.5, mid - lo * mid * 0.92 + 0.5);
83+ }
84+ g.stroke();
85+
86+ for (let i = 0; i < this.markers.length; i++) {
87+ const m = this.markers[i];
88+ const x = (m.t / dur) * w;
89+ g.strokeStyle = this.color;
90+ g.lineWidth = i === this.selected ? 2.5 : 1.2;
91+ g.globalAlpha = m.excluded ? 0.35 : 1;
92+ g.beginPath();
93+ g.moveTo(x, 2);
94+ g.lineTo(x, h - 2);
95+ g.stroke();
96+ g.beginPath();
97+ g.arc(x, 8, 4.5, 0, 2 * Math.PI);
98+ if (m.excluded) g.stroke();
99+ else { g.fillStyle = this.color; g.fill(); }
100+ g.globalAlpha = 1;
101+ }
102+ }
103+}
tsconfig.jsonadded+15−0View file
@@ -0,0 +1,15 @@
1+{
2+ "compilerOptions": {
3+ "target": "ES2022",
4+ "module": "ESNext",
5+ "moduleResolution": "bundler",
6+ "allowImportingTsExtensions": true,
7+ "resolveJsonModule": true,
8+ "strict": true,
9+ "noUncheckedIndexedAccess": false,
10+ "noEmit": true,
11+ "skipLibCheck": true,
12+ "types": ["vite/client", "node"]
13+ },
14+ "include": ["src", "scripts"]
15+}
vite.config.tsadded+6−0View file
@@ -0,0 +1,6 @@
1+import { defineConfig } from 'vite';
2+
3+export default defineConfig({
4+ base: './',
5+ build: { target: 'es2022' },
6+});