Editable persistent trainings, multi-select exclusion, playback pulses, instrawave bindings
Training clips now persist (audio in IndexedDB, detections and features in
localStorage) and reopen in the capture editor, which gains rubber-band and
shift-click multi-select with X to toggle exclusion. Timeline hits flash
with strength-scaled rings as the playhead crosses them. Each voice binds
to a synth preset or an instrawave instrument with per-voice tuning
sliders; bakes are computed on demand from a bake bar, voices sharing an
instrument share its bake and render in one event-list pass, and the GPU
is only touched when an instrawave binding is present.
14 changed files+893−645
README.mdmodified+39−18View file
@@ -21,10 +21,13 @@ while the ordering is preserved.
2121 A voice is a named class of sound plus its training examples. Recording
2222 into a voice runs an onset detector (spectral flux against a local mean,
2323 with a sensitivity control, since a soft syllable and a sharp tap sit at
24-very different flux levels) and shows the detected hits as markers on the
25-waveform; each can be auditioned and excluded before the batch is
26-committed, so a cough or a stray detection is visible rather than silently
27-poisoning the label. Each committed hit is stored as a small feature
24+very different flux levels) and opens the clip in an editor: the detected
25+hits are markers on the waveform, auditioned by click, selected in groups
26+by dragging or shift-clicking, and excluded with a button or the X key, so
27+a cough or a stray detection is visible rather than silently poisoning the
28+label. Clips persist (audio in IndexedDB, detections and features in
29+localStorage) and reopen in the same editor later; deleting a clip removes
30+its hits from the voice. Each committed hit is stored as a small feature
2831 vector: spectral centroid and rolloff, band energy ratios, zero-crossing
2932 rate, attack and decay times. Loudness is deliberately kept out of this
3033 vector; it is expression, not identity, so playing softly does not change
@@ -38,21 +41,35 @@ be wrong, and the fix is one gesture: drag the marker to the right lane.
3841 "Learn" folds the corrected hit back into that voice's training set, so
3942 the labelling improves with use; there is no model to refit.
4043
41-Voices and their training persist in localStorage. Takes do not; the take
42-is a performance, and re-recording one is the cheap operation.
44+Takes are not persisted; the take is a performance, and re-recording one
45+is the cheap operation. During playback each hit flashes as the playhead
46+crosses it, ring size following the hit's strength, so the dynamics are
47+visible in motion as well as in the bar heights.
4348
4449 ## Rendering
4550
46-The seam is `src/render/types.ts`: events in, audio out. The current
47-renderer is a set of placeholder WebAudio drum synths (kick, snare, hat,
48-tom) so the whole capture-classify-arrange loop is playable end to end.
49-The intended renderer is [instrawave](../instrawave), where a binding
50-becomes an instrument file plus parameter values and each hit becomes a
51-stimulus event `{t, set: {vel: ...}}`. Because an instrawave bake depends
52-only on geometry and accuracy, voices that share a bake and differ only in
53-tuning (a low, wide, centre-struck drum against a high edge strike) render
54-in a single pass; the renderer will group events by bake key and mix one
55-trace per group. Loudness-to-velocity is the only feature mapping for now.
51+The seam is `src/render/types.ts`: events in, audio out. Each voice is
52+bound either to a placeholder WebAudio synth preset (kick, snare, hat,
53+tom) or to an [instrawave](../instrawave) instrument (drum, dulcimer,
54+chime) with per-voice tuning overrides, edited as sliders under the voice.
55+Playback touches the GPU only when at least one bound instrument is
56+instrawave; a purely synth arrangement never asks for a device.
57+
58+The instrawave instruments are held at their default geometry and
59+accuracy, so each has exactly one bake, computed on demand from the bake
60+bar (with progress and a cancel) and kept for the session. All per-voice
61+character travels as tuning, which is free: every voice bound to the same
62+instrument shares its bake, and one render call carries all their events,
63+each hit an event `{t, set: {...tuning, vel}}` with the voice's overrides
64+and its strength scaling the excitation (`vel`, or `amp` for the plucked
65+dulcimer; the equations are linear, so this is a pure scale). Traces come
66+back at the solver rate, are resampled by linear interpolation, levelled
67+per instrument (a pressure trace carries no absolute scale worth keeping
68+against a synth's), and mixed; relative volumes within one instrument stay
69+physical. Mixes are cached against the exact events and bindings, so
70+replaying an unchanged take is free.
71+
72+Loudness to excitation strength is the only feature mapping for now.
5673 Duration to decay, vocal pitch to drum pitch, and brightness to strike
5774 position are natural next steps and the event structure leaves room for
5875 them.
@@ -67,8 +84,12 @@ unfamiliar sound lands on whatever is closest. Overlapping hits closer
6784 than 50 ms merge into one. The browser's microphone processing is asked to
6885 stand down (automatic gain control would rewrite the very dynamics being
6986 tracked), but a system-level AGC outside the browser will still flatten
70-them. Training audio is not persisted, only its features, so trained hits
71-cannot be re-auditioned after a reload.
87+them. Bakes are not persisted (the drum's is a couple of hundred MB), so
88+the first instrawave playback of a session pays the precompute, about 8 s
89+for the drum on the development machine. A hit learned from a take stores
90+only its features, so it cannot be re-auditioned later; clips can. A
91+multi-event instrawave render costs one synthesis pass per event over the
92+whole trace, so a very dense take renders slowly.
7293
7394 ## Running it
7495
index.htmlmodified+17−5View file
@@ -31,7 +31,16 @@
3131 .voice-row:nth-child(odd) { background: rgba(128, 128, 128, 0.09); }
3232 .voice-row .swatch { width: 0.8rem; height: 0.8rem; border-radius: 3px; flex: none; }
3333 .voice-row input.name { width: 8rem; border: none; background: transparent; font-weight: 600; }
34- .voice-row .count { opacity: 0.65; font-size: 0.8rem; min-width: 5.5rem; }
34+ .voice-row .count { opacity: 0.65; font-size: 0.8rem; min-width: 7.5rem; }
35+ .tuning-panel {
36+ display: grid; grid-template-columns: repeat(auto-fill, minmax(15rem, 1fr));
37+ gap: 0.2rem 1.5rem; padding: 0.3rem 0.6rem 0.5rem 1.8rem; font-size: 0.82rem;
38+ }
39+ .tuning-panel .param { display: grid; }
40+ .tuning-panel label { display: flex; justify-content: space-between; }
41+ .tuning-panel label span:last-child { opacity: 0.7; font-variant-numeric: tabular-nums; }
42+ .tuning-panel input[type='range'] { width: 100%; }
43+ #bakebar progress { width: 8rem; }
3544 .voice-row .spacer { flex: 1; }
3645 .rec { color: #d33; }
3746 canvas.wave { width: 100%; height: 110px; border-radius: 6px; display: block; touch-action: none; }
@@ -51,7 +60,7 @@
5160 hits are classified back onto the voices and rendered on instruments.
5261 </p>
5362
54- <h2>Voices <span class="cost">record several examples of each sound: a syllable, a tap, anything repeatable</span></h2>
63+ <h2>Voices <span class="cost">record several examples of each sound; recordings stay editable</span></h2>
5564 <div id="voice-list"></div>
5665 <div class="bar">
5766 <input id="voice-name" type="text" placeholder="new voice name (e.g. doo)" />
@@ -63,15 +72,17 @@
6372 <div id="capture" hidden>
6473 <div class="bar">
6574 <strong id="capture-title"></strong>
66- <span class="hint">click a marker to audition; exclude anything that is not the sound</span>
75+ <select id="capture-clip"></select>
76+ <span class="hint">click a hit to audition · drag or shift-click to select several · X toggles exclusion</span>
6777 </div>
6878 <canvas id="capture-wave" class="wave"></canvas>
6979 <div class="bar">
7080 <label>sensitivity <input id="capture-sens" type="range" min="0" max="1" step="0.01" value="0.6" /></label>
7181 <button id="capture-exclude" disabled>Exclude</button>
82+ <span id="capture-count" class="hint"></span>
7283 <span style="flex:1"></span>
73- <button id="capture-commit"></button>
74- <button id="capture-discard">Discard</button>
84+ <button id="capture-delete">Delete clip</button>
85+ <button id="capture-done">Done</button>
7586 </div>
7687 </div>
7788
@@ -84,6 +95,7 @@
8495 <span style="flex:1"></span>
8596 <button id="take-play" disabled>Play</button>
8697 </div>
98+ <div class="bar" id="bakebar" hidden></div>
8799 <canvas id="timeline" class="tl"></canvas>
88100 <div class="bar">
89101 <span class="hint">drag a hit to another lane to reassign · double-click a lane to add a hit · delete key removes</span>
package-lock.jsonmodified+8−458View file
@@ -13,6 +13,7 @@
1313 },
1414 "devDependencies": {
1515 "@types/node": "^26.1.1",
16+ "@webgpu/types": "^0.1.44",
1617 "typescript": "^5.5.0",
1718 "vite": "^5.4.0",
1819 "vite-node": "^6.0.0"
@@ -327,22 +328,6 @@
327328 "node": ">=12"
328329 }
329330 },
330- "node_modules/@esbuild/netbsd-arm64": {
331- "version": "0.28.2",
332- "resolved": "https://registry.npmjs.org/@esbuild/netbsd-arm64/-/netbsd-arm64-0.28.2.tgz",
333- "integrity": "sha512-sSATRjPeDBg3pdgHoQfoYBob11Kk1FGa9lui5RIHZCoCkJa9QKlvl3/vKz2usCmYYjs7ymJR/2Nnsqe+Hjt5nw==",
334- "cpu": [
335- "arm64"
336- ],
337- "extraneous": true,
338- "license": "MIT",
339- "os": [
340- "netbsd"
341- ],
342- "engines": {
343- "node": ">=18"
344- }
345- },
346331 "node_modules/@esbuild/netbsd-x64": {
347332 "version": "0.21.5",
348333 "resolved": "https://registry.npmjs.org/@esbuild/netbsd-x64/-/netbsd-x64-0.21.5.tgz",
@@ -360,22 +345,6 @@
360345 "node": ">=12"
361346 }
362347 },
363- "node_modules/@esbuild/openbsd-arm64": {
364- "version": "0.28.2",
365- "resolved": "https://registry.npmjs.org/@esbuild/openbsd-arm64/-/openbsd-arm64-0.28.2.tgz",
366- "integrity": "sha512-AL2qJILH7lNjrDmCQDvdxMfAUIv8KMNZOvrwAQ8i8//ntL9FflhOyMJ8OZSMBb8/AWXe3/5v5S20y3zCoZWKoQ==",
367- "cpu": [
368- "arm64"
369- ],
370- "extraneous": true,
371- "license": "MIT",
372- "os": [
373- "openbsd"
374- ],
375- "engines": {
376- "node": ">=18"
377- }
378- },
379348 "node_modules/@esbuild/openbsd-x64": {
380349 "version": "0.21.5",
381350 "resolved": "https://registry.npmjs.org/@esbuild/openbsd-x64/-/openbsd-x64-0.21.5.tgz",
@@ -393,22 +362,6 @@
393362 "node": ">=12"
394363 }
395364 },
396- "node_modules/@esbuild/openharmony-arm64": {
397- "version": "0.28.2",
398- "resolved": "https://registry.npmjs.org/@esbuild/openharmony-arm64/-/openharmony-arm64-0.28.2.tgz",
399- "integrity": "sha512-WkhYDmpTjLvGlScA1rwjRUmhl4k8oXR3cIbtqWmELgU/dFeHHlEllxDvdWcNJV9rbzCexB5vz8gtNewWLgCT7Q==",
400- "cpu": [
401- "arm64"
402- ],
403- "extraneous": true,
404- "license": "MIT",
405- "os": [
406- "openharmony"
407- ],
408- "engines": {
409- "node": ">=18"
410- }
411- },
412365 "node_modules/@esbuild/sunos-x64": {
413366 "version": "0.21.5",
414367 "resolved": "https://registry.npmjs.org/@esbuild/sunos-x64/-/sunos-x64-0.21.5.tgz",
@@ -1193,6 +1146,13 @@
11931146 "undici-types": "~8.3.0"
11941147 }
11951148 },
1149+ "node_modules/@webgpu/types": {
1150+ "version": "0.1.72",
1151+ "resolved": "https://registry.npmjs.org/@webgpu/types/-/types-0.1.72.tgz",
1152+ "integrity": "sha512-0cF7RFM2edNoiIS1ODJp0/Gzv4/xSXhwoR0YCza+OWpJWtn4wmo9DvK91aLlH9+uUnwIriP7ZiC3WitmyhuzBw==",
1153+ "dev": true,
1154+ "license": "BSD-3-Clause"
1155+ },
11961156 "node_modules/cac": {
11971157 "version": "7.0.0",
11981158 "resolved": "https://registry.npmjs.org/cac/-/cac-7.0.0.tgz",
@@ -1869,416 +1829,6 @@
18691829 "url": "https://opencollective.com/antfu"
18701830 }
18711831 },
1872- "node_modules/vite-node/node_modules/@esbuild/aix-ppc64": {
1873- "version": "0.28.2",
1874- "resolved": "https://registry.npmjs.org/@esbuild/aix-ppc64/-/aix-ppc64-0.28.2.tgz",
1875- "integrity": "sha512-XExcO+dvLKvVtNTibSTBej1NCAbaGhWn9Ww1ZPx80qsahhPFe/8jgWP0IchNe0F3HwkU7n8ejhH8bjonqht8mQ==",
1876- "cpu": [
1877- "ppc64"
1878- ],
1879- "extraneous": true,
1880- "license": "MIT",
1881- "os": [
1882- "aix"
1883- ],
1884- "engines": {
1885- "node": ">=18"
1886- }
1887- },
1888- "node_modules/vite-node/node_modules/@esbuild/android-arm": {
1889- "version": "0.28.2",
1890- "resolved": "https://registry.npmjs.org/@esbuild/android-arm/-/android-arm-0.28.2.tgz",
1891- "integrity": "sha512-kXXoiPVVGQcnIYGOeaovwOURpniDBpSq4A03qkQ+BMQqtGG6HYap3xne9C1O1yo4TR3qxlCX5IqqmX6fFo2Lqg==",
1892- "cpu": [
1893- "arm"
1894- ],
1895- "extraneous": true,
1896- "license": "MIT",
1897- "os": [
1898- "android"
1899- ],
1900- "engines": {
1901- "node": ">=18"
1902- }
1903- },
1904- "node_modules/vite-node/node_modules/@esbuild/android-arm64": {
1905- "version": "0.28.2",
1906- "resolved": "https://registry.npmjs.org/@esbuild/android-arm64/-/android-arm64-0.28.2.tgz",
1907- "integrity": "sha512-5YfKeeI8qWfBZIX+u2xZC3Zlb3Os/gLS2sbEKM+I4ZOcsWmHS2WLysCcQZDAFRslDUU5Oiq44gf6PYN1vGwG5A==",
1908- "cpu": [
1909- "arm64"
1910- ],
1911- "extraneous": true,
1912- "license": "MIT",
1913- "os": [
1914- "android"
1915- ],
1916- "engines": {
1917- "node": ">=18"
1918- }
1919- },
1920- "node_modules/vite-node/node_modules/@esbuild/android-x64": {
1921- "version": "0.28.2",
1922- "resolved": "https://registry.npmjs.org/@esbuild/android-x64/-/android-x64-0.28.2.tgz",
1923- "integrity": "sha512-O387ite7SzUyCcy3JQX4P4bLtEA7bLLkx+esve5JHnyYfNTxcVpXZo9jhdB0lTKN44gztELTdU7nS8Nr16Fs1Q==",
1924- "cpu": [
1925- "x64"
1926- ],
1927- "extraneous": true,
1928- "license": "MIT",
1929- "os": [
1930- "android"
1931- ],
1932- "engines": {
1933- "node": ">=18"
1934- }
1935- },
1936- "node_modules/vite-node/node_modules/@esbuild/darwin-arm64": {
1937- "version": "0.28.2",
1938- "resolved": "https://registry.npmjs.org/@esbuild/darwin-arm64/-/darwin-arm64-0.28.2.tgz",
1939- "integrity": "sha512-n4KqkOQrraxHJcgjM1RvwbigfQKIKJVpM7xp+KsxiyUSrRdIXnt73VhrPAx0fV44hgfmIVKjxMN9J1t5jySVkw==",
1940- "cpu": [
1941- "arm64"
1942- ],
1943- "extraneous": true,
1944- "license": "MIT",
1945- "os": [
1946- "darwin"
1947- ],
1948- "engines": {
1949- "node": ">=18"
1950- }
1951- },
1952- "node_modules/vite-node/node_modules/@esbuild/darwin-x64": {
1953- "version": "0.28.2",
1954- "resolved": "https://registry.npmjs.org/@esbuild/darwin-x64/-/darwin-x64-0.28.2.tgz",
1955- "integrity": "sha512-uq6suIWYP37qzGddBKPw5QEQPi6HiLGsO7UmkpfyaYNQ3D+rN6w6WfwH+nuqcGXWvawGwxOEroO4YGnFh95azw==",
1956- "cpu": [
1957- "x64"
1958- ],
1959- "extraneous": true,
1960- "license": "MIT",
1961- "os": [
1962- "darwin"
1963- ],
1964- "engines": {
1965- "node": ">=18"
1966- }
1967- },
1968- "node_modules/vite-node/node_modules/@esbuild/freebsd-arm64": {
1969- "version": "0.28.2",
1970- "resolved": "https://registry.npmjs.org/@esbuild/freebsd-arm64/-/freebsd-arm64-0.28.2.tgz",
1971- "integrity": "sha512-n+I0BTSRIoy+d6RPKnEVwql5UwBJolytvY4mAOIEJorKlqgPII8ix6slVVrfZ5Tnj7glIZvloylbB/EJPMWEXw==",
1972- "cpu": [
1973- "arm64"
1974- ],
1975- "extraneous": true,
1976- "license": "MIT",
1977- "os": [
1978- "freebsd"
1979- ],
1980- "engines": {
1981- "node": ">=18"
1982- }
1983- },
1984- "node_modules/vite-node/node_modules/@esbuild/freebsd-x64": {
1985- "version": "0.28.2",
1986- "resolved": "https://registry.npmjs.org/@esbuild/freebsd-x64/-/freebsd-x64-0.28.2.tgz",
1987- "integrity": "sha512-78XJTJkvPs0kz2w61301PJjXl4g7q3JqiYMZ/M/yVI73EHBrCRTgkhu9oqG7vPqq+a/yadEW8aD+agKlk5xrmg==",
1988- "cpu": [
1989- "x64"
1990- ],
1991- "extraneous": true,
1992- "license": "MIT",
1993- "os": [
1994- "freebsd"
1995- ],
1996- "engines": {
1997- "node": ">=18"
1998- }
1999- },
2000- "node_modules/vite-node/node_modules/@esbuild/linux-arm": {
2001- "version": "0.28.2",
2002- "resolved": "https://registry.npmjs.org/@esbuild/linux-arm/-/linux-arm-0.28.2.tgz",
2003- "integrity": "sha512-XlDnu2q5yoqems+xay6wSAcg9DDD7K9RLKZEBOMZm3ckNpJBvOX20tSfby8KfrrhINDyv9V2YVZKY/SpoGJI8w==",
2004- "cpu": [
2005- "arm"
2006- ],
2007- "extraneous": true,
2008- "license": "MIT",
2009- "os": [
2010- "linux"
2011- ],
2012- "engines": {
2013- "node": ">=18"
2014- }
2015- },
2016- "node_modules/vite-node/node_modules/@esbuild/linux-arm64": {
2017- "version": "0.28.2",
2018- "resolved": "https://registry.npmjs.org/@esbuild/linux-arm64/-/linux-arm64-0.28.2.tgz",
2019- "integrity": "sha512-pW4AC0P3it8c7do9MVM4p51FzHzdM/TZrerurgRcHJ2WTa1VQ1CIq18xncfpBJw4ojkiZZrKW2yIBWBP92j6Ug==",
2020- "cpu": [
2021- "arm64"
2022- ],
2023- "extraneous": true,
2024- "license": "MIT",
2025- "os": [
2026- "linux"
2027- ],
2028- "engines": {
2029- "node": ">=18"
2030- }
2031- },
2032- "node_modules/vite-node/node_modules/@esbuild/linux-ia32": {
2033- "version": "0.28.2",
2034- "resolved": "https://registry.npmjs.org/@esbuild/linux-ia32/-/linux-ia32-0.28.2.tgz",
2035- "integrity": "sha512-CYbnj78HsIeA+DhgUKgFCfvNsTHFhMMrinUrMZpDXJXKN8T3XViTZ/+wtHeVxEWY8ewSzTFN+nRmSwO2tZaLUQ==",
2036- "cpu": [
2037- "ia32"
2038- ],
2039- "extraneous": true,
2040- "license": "MIT",
2041- "os": [
2042- "linux"
2043- ],
2044- "engines": {
2045- "node": ">=18"
2046- }
2047- },
2048- "node_modules/vite-node/node_modules/@esbuild/linux-loong64": {
2049- "version": "0.28.2",
2050- "resolved": "https://registry.npmjs.org/@esbuild/linux-loong64/-/linux-loong64-0.28.2.tgz",
2051- "integrity": "sha512-buwkd8nsph4R+ajRvw0qM5Hja/TXQow3ptzWO2EbG/cqcIkHloRrdlBtQlshyYGTNFvfkfJ5tpPLVkY4DtsPfQ==",
2052- "cpu": [
2053- "loong64"
2054- ],
2055- "extraneous": true,
2056- "license": "MIT",
2057- "os": [
2058- "linux"
2059- ],
2060- "engines": {
2061- "node": ">=18"
2062- }
2063- },
2064- "node_modules/vite-node/node_modules/@esbuild/linux-mips64el": {
2065- "version": "0.28.2",
2066- "resolved": "https://registry.npmjs.org/@esbuild/linux-mips64el/-/linux-mips64el-0.28.2.tgz",
2067- "integrity": "sha512-ZVykbDyk7519VwiNb9Lcj9m8XM6v5V9uKPvrEMkkEedVewf+0itkhahp4HDpgERXhwLRpWFypsGbG/J8s0QjJA==",
2068- "cpu": [
2069- "mips64el"
2070- ],
2071- "extraneous": true,
2072- "license": "MIT",
2073- "os": [
2074- "linux"
2075- ],
2076- "engines": {
2077- "node": ">=18"
2078- }
2079- },
2080- "node_modules/vite-node/node_modules/@esbuild/linux-ppc64": {
2081- "version": "0.28.2",
2082- "resolved": "https://registry.npmjs.org/@esbuild/linux-ppc64/-/linux-ppc64-0.28.2.tgz",
2083- "integrity": "sha512-CAXl+Dtd9UUuJd8pKKdwh6MLm3MUMiqMPmhZ3tTSXPqfyQ3vDl6R5hZdZ/kYojK4ofXtdfSv1tFq8XzWx3heNQ==",
2084- "cpu": [
2085- "ppc64"
2086- ],
2087- "extraneous": true,
2088- "license": "MIT",
2089- "os": [
2090- "linux"
2091- ],
2092- "engines": {
2093- "node": ">=18"
2094- }
2095- },
2096- "node_modules/vite-node/node_modules/@esbuild/linux-riscv64": {
2097- "version": "0.28.2",
2098- "resolved": "https://registry.npmjs.org/@esbuild/linux-riscv64/-/linux-riscv64-0.28.2.tgz",
2099- "integrity": "sha512-GeXCej4IQtU1B+QlDV8W/RRvbzI3O/Stss+/bCXv4lZls5WGRtu2a+3JkA3i4qIUlMXpcHebWpF8AkJhATowuA==",
2100- "cpu": [
2101- "riscv64"
2102- ],
2103- "extraneous": true,
2104- "license": "MIT",
2105- "os": [
2106- "linux"
2107- ],
2108- "engines": {
2109- "node": ">=18"
2110- }
2111- },
2112- "node_modules/vite-node/node_modules/@esbuild/linux-s390x": {
2113- "version": "0.28.2",
2114- "resolved": "https://registry.npmjs.org/@esbuild/linux-s390x/-/linux-s390x-0.28.2.tgz",
2115- "integrity": "sha512-3H1weTYZPxt/WOhByszQZybS9w5lKzUn1FDMsgEChbHWQwHYQQRfBxgCcZvPhjHfKyJjIievvMmEUawJrdY9Dg==",
2116- "cpu": [
2117- "s390x"
2118- ],
2119- "extraneous": true,
2120- "license": "MIT",
2121- "os": [
2122- "linux"
2123- ],
2124- "engines": {
2125- "node": ">=18"
2126- }
2127- },
2128- "node_modules/vite-node/node_modules/@esbuild/linux-x64": {
2129- "version": "0.28.2",
2130- "resolved": "https://registry.npmjs.org/@esbuild/linux-x64/-/linux-x64-0.28.2.tgz",
2131- "integrity": "sha512-4xTZr1FUmSoQW4XIWmit3tzQrUTZM+N3P0XV8xROKYF50XfI7xeO90+1bZvNwxIufQ9hDQVRJH5YhgPVF8A/HQ==",
2132- "cpu": [
2133- "x64"
2134- ],
2135- "extraneous": true,
2136- "license": "MIT",
2137- "os": [
2138- "linux"
2139- ],
2140- "engines": {
2141- "node": ">=18"
2142- }
2143- },
2144- "node_modules/vite-node/node_modules/@esbuild/netbsd-x64": {
2145- "version": "0.28.2",
2146- "resolved": "https://registry.npmjs.org/@esbuild/netbsd-x64/-/netbsd-x64-0.28.2.tgz",
2147- "integrity": "sha512-lqnzCV+mM0gIADaKihiCg6ifgfU2L3h5E33rNQBN1Y4MaVGnzryzmvvf7UHxprpQdE8hpqLolJ9Rl+SkIRDpyw==",
2148- "cpu": [
2149- "x64"
2150- ],
2151- "extraneous": true,
2152- "license": "MIT",
2153- "os": [
2154- "netbsd"
2155- ],
2156- "engines": {
2157- "node": ">=18"
2158- }
2159- },
2160- "node_modules/vite-node/node_modules/@esbuild/openbsd-x64": {
2161- "version": "0.28.2",
2162- "resolved": "https://registry.npmjs.org/@esbuild/openbsd-x64/-/openbsd-x64-0.28.2.tgz",
2163- "integrity": "sha512-QtiuPytchRyC4rwUKhexJdQKvDuZ6hWloi3igqPQNUJCS1/v9EiO3UTOXR6A3FoMo4fnAKbWJdqaIwhOzh8qEw==",
2164- "cpu": [
2165- "x64"
2166- ],
2167- "extraneous": true,
2168- "license": "MIT",
2169- "os": [
2170- "openbsd"
2171- ],
2172- "engines": {
2173- "node": ">=18"
2174- }
2175- },
2176- "node_modules/vite-node/node_modules/@esbuild/sunos-x64": {
2177- "version": "0.28.2",
2178- "resolved": "https://registry.npmjs.org/@esbuild/sunos-x64/-/sunos-x64-0.28.2.tgz",
2179- "integrity": "sha512-GPMSkTOtMnv2U2F8gxe4Io6qmVs+YKyp832Etqqxr0hFngmXQ3rzwytelm3GIn7T4VviRUlf3sOgBOiTdvaf7g==",
2180- "cpu": [
2181- "x64"
2182- ],
2183- "extraneous": true,
2184- "license": "MIT",
2185- "os": [
2186- "sunos"
2187- ],
2188- "engines": {
2189- "node": ">=18"
2190- }
2191- },
2192- "node_modules/vite-node/node_modules/@esbuild/win32-arm64": {
2193- "version": "0.28.2",
2194- "resolved": "https://registry.npmjs.org/@esbuild/win32-arm64/-/win32-arm64-0.28.2.tgz",
2195- "integrity": "sha512-PIhhEkE9uPBleRBrQEJpUn7MBnibZzbGzYWPmY3x+YoVg/95zbjB4CxPPOQ8l5tYYM4mMaCthF8/1DIfBQQyWQ==",
2196- "cpu": [
2197- "arm64"
2198- ],
2199- "extraneous": true,
2200- "license": "MIT",
2201- "os": [
2202- "win32"
2203- ],
2204- "engines": {
2205- "node": ">=18"
2206- }
2207- },
2208- "node_modules/vite-node/node_modules/@esbuild/win32-ia32": {
2209- "version": "0.28.2",
2210- "resolved": "https://registry.npmjs.org/@esbuild/win32-ia32/-/win32-ia32-0.28.2.tgz",
2211- "integrity": "sha512-YmJbfTlvU7Sdn9BB+4PRES4oB6pxgS37MAONj+hBr/cpXS1aBPKXxNnDbu+QCWPj0o9dgyxeq79g6c5P8KeuYA==",
2212- "cpu": [
2213- "ia32"
2214- ],
2215- "extraneous": true,
2216- "license": "MIT",
2217- "os": [
2218- "win32"
2219- ],
2220- "engines": {
2221- "node": ">=18"
2222- }
2223- },
2224- "node_modules/vite-node/node_modules/@esbuild/win32-x64": {
2225- "version": "0.28.2",
2226- "resolved": "https://registry.npmjs.org/@esbuild/win32-x64/-/win32-x64-0.28.2.tgz",
2227- "integrity": "sha512-5ebpxr3nWMzrL/rnUI755Jkuee0bHL/Gq0WTF9lvcpv73wAp5eu8MfBUgWK9bhWvZjj7yX8etf/8tI8Ney695g==",
2228- "cpu": [
2229- "x64"
2230- ],
2231- "extraneous": true,
2232- "license": "MIT",
2233- "os": [
2234- "win32"
2235- ],
2236- "engines": {
2237- "node": ">=18"
2238- }
2239- },
2240- "node_modules/vite-node/node_modules/esbuild": {
2241- "version": "0.28.2",
2242- "resolved": "https://registry.npmjs.org/esbuild/-/esbuild-0.28.2.tgz",
2243- "integrity": "sha512-HKVLS8dvII+xoKW9kmqxbRKrnWEXfJJr/FZhhJmiqIB0e053QNYFqOBouTMO/k5sID4MvCiUCvv8b9M4h32wIA==",
2244- "extraneous": true,
2245- "hasInstallScript": true,
2246- "license": "MIT",
2247- "bin": {
2248- "esbuild": "bin/esbuild"
2249- },
2250- "engines": {
2251- "node": ">=18"
2252- },
2253- "optionalDependencies": {
2254- "@esbuild/aix-ppc64": "0.28.2",
2255- "@esbuild/android-arm": "0.28.2",
2256- "@esbuild/android-arm64": "0.28.2",
2257- "@esbuild/android-x64": "0.28.2",
2258- "@esbuild/darwin-arm64": "0.28.2",
2259- "@esbuild/darwin-x64": "0.28.2",
2260- "@esbuild/freebsd-arm64": "0.28.2",
2261- "@esbuild/freebsd-x64": "0.28.2",
2262- "@esbuild/linux-arm": "0.28.2",
2263- "@esbuild/linux-arm64": "0.28.2",
2264- "@esbuild/linux-ia32": "0.28.2",
2265- "@esbuild/linux-loong64": "0.28.2",
2266- "@esbuild/linux-mips64el": "0.28.2",
2267- "@esbuild/linux-ppc64": "0.28.2",
2268- "@esbuild/linux-riscv64": "0.28.2",
2269- "@esbuild/linux-s390x": "0.28.2",
2270- "@esbuild/linux-x64": "0.28.2",
2271- "@esbuild/netbsd-arm64": "0.28.2",
2272- "@esbuild/netbsd-x64": "0.28.2",
2273- "@esbuild/openbsd-arm64": "0.28.2",
2274- "@esbuild/openbsd-x64": "0.28.2",
2275- "@esbuild/openharmony-arm64": "0.28.2",
2276- "@esbuild/sunos-x64": "0.28.2",
2277- "@esbuild/win32-arm64": "0.28.2",
2278- "@esbuild/win32-ia32": "0.28.2",
2279- "@esbuild/win32-x64": "0.28.2"
2280- }
2281- },
22821832 "node_modules/vite-node/node_modules/vite": {
22831833 "version": "8.2.2",
22841834 "resolved": "https://registry.npmjs.org/vite/-/vite-8.2.2.tgz",
package.jsonmodified+3−2View file
@@ -19,6 +19,7 @@
1919 "@types/node": "^26.1.1",
2020 "typescript": "^5.5.0",
2121 "vite": "^5.4.0",
22- "vite-node": "^6.0.0"
22+ "vite-node": "^6.0.0",
23+ "@webgpu/types": "^0.1.44"
2324 }
24-}
25+}
scripts/test-node.tsmodified+17−0View file
@@ -8,6 +8,7 @@ import { detectOnsets } from '../src/analysis/onsets.ts';
88 import { extractHit } from '../src/analysis/features.ts';
99 import { classify, type TrainingSet } from '../src/analysis/classify.ts';
1010 import { gainFromPeakDb } from '../src/render/types.ts';
11+import { mixInto, peak, resampleLinear } from '../src/render/mix.ts';
1112
1213 const SR = 48000;
1314 let failures = 0;
@@ -92,5 +93,21 @@ function burst(x: Float32Array, t0: number, freq: number, decay: number, amp: nu
9293 check(gainFromPeakDb(-90, -10) > 0.15, 'gain: very soft hits stay audible');
9394 }
9495
96+// --- mix helpers ---------------------------------------------------------------
97+
98+{
99+ const tone = new Float32Array(1000);
100+ for (let i = 0; i < tone.length; i++) tone[i] = Math.sin((2 * Math.PI * 40 * i) / 1000);
101+ const down = resampleLinear(tone, 96000, 48000);
102+ check(down.length === 500, `resample: 1000 @ 96k -> ${down.length} @ 48k`);
103+ let worst = 0;
104+ for (let i = 0; i < down.length; i++) worst = Math.max(worst, Math.abs(down[i] - tone[2 * i]));
105+ check(worst < 5e-3, `resample: worst deviation ${worst.toExponential(1)} on a low tone`);
106+ const a = new Float32Array([1, 2, 3]);
107+ mixInto(a, new Float32Array([1, 1, 1, 9]), 0.5);
108+ check(a[0] === 1.5 && a[2] === 3.5, 'mixInto: sums with gain, clipped to dst length');
109+ check(Math.abs(peak(new Float32Array([0.1, -0.9, 0.4])) - 0.9) < 1e-6, 'peak: absolute maximum');
110+}
111+
95112 console.log(failures ? `\n${failures} failure(s)` : '\nall checks passed');
96113 process.exit(failures ? 1 : 0);
src/main.tsmodified+429−127View file
@@ -1,8 +1,11 @@
11 /**
2- * The page: voices (named, trained by example), a take (recorded once,
3- * segmented, classified onto the voices), a timeline (the take as editable
4- * events), and playback through the renderer seam in render/types.ts. All
5- * analysis is offline on the recorded clip; nothing here is realtime.
2+ * The page: voices (named, trained by example, each bound to a synth
3+ * preset or an instrawave instrument), a take (recorded once, segmented,
4+ * classified onto the voices), a timeline (the take as editable events),
5+ * and playback through the renderer seam in render/types.ts. Training
6+ * clips persist (audio in IndexedDB, detections and features in
7+ * localStorage) and reopen in the same editor they were made in. The GPU
8+ * is touched only when at least one bound instrument is instrawave.
69 */
710 import type { Clip } from './audio/recorder.ts';
811 import { Recorder } from './audio/recorder.ts';
@@ -11,15 +14,19 @@ import { extractHit } from './analysis/features.ts';
1114 import { classify, type TrainingSet } from './analysis/classify.ts';
1215 import { renderSynth } from './render/synth.ts';
1316 import { gainFromPeakDb, type NoteEvent, type SynthPreset } from './render/types.ts';
17+import { mixInto, peak, resampleLinear } from './render/mix.ts';
18+import { IwEngine, PhaseCancelled, type StimulusEvent } from './render/instrawave.ts';
19+import {
20+ clearClipAudio, clearStore, deleteClipAudio, getClipAudio, loadVoices, putClipAudio,
21+ saveVoices, trainPool, type TrainClip, type Voice,
22+} from './store.ts';
1423 import { WaveView, type Marker } from './ui/waveform.ts';
1524 import { Timeline } from './ui/timeline.ts';
1625
1726 const PALETTE = ['#e4573d', '#3d7de4', '#3dbb6e', '#d9a520', '#9061d9', '#d95fa8', '#4fb8c9', '#96a13a'];
1827 const PRESETS: SynthPreset[] = ['kick', 'snare', 'hat', 'tom'];
19-const STORE_KEY = 'instrawave-compose/v1';
28+const MIX_SR = 44100;
2029
21-interface TrainHit { vec: number[]; peakDb: number; }
22-interface Voice { id: string; name: string; color: string; preset: SynthPreset; train: TrainHit[]; }
2330 interface TakeHit {
2431 id: string; t: number; voiceId: string; gain: number;
2532 peakDb: number; vec: number[] | null; // null for hand-added hits
@@ -33,39 +40,32 @@ const uid = () => Math.random().toString(36).slice(2, 10);
3340
3441 let voices: Voice[] = loadVoices();
3542 let take: { clip: Clip | null; hits: TakeHit[] } = { clip: null, hits: [] };
36-let pending: { clip: Clip; voiceId: string; onsets: number[]; excluded: Set<number> } | null = null;
43+let edit: { voiceId: string; clipId: string; samples: Float32Array; sr: number } | null = null;
3744 let recTarget: { kind: 'voice'; voiceId: string } | { kind: 'take' } | null = null;
45+const expandedTuning = new Set<string>();
3846
3947 let actx: AudioContext | null = null;
4048 const audio = (): AudioContext => (actx ??= new AudioContext());
4149 let recorder: Recorder | null = null;
4250 let playing: { src: AudioBufferSourceNode; t0: number; dur: number } | null = null;
51+let renderCache: { sig: string; buf: AudioBuffer } | null = null;
4352
44-function loadVoices(): Voice[] {
45- try {
46- const raw = localStorage.getItem(STORE_KEY);
47- if (raw) return (JSON.parse(raw) as { voices: Voice[] }).voices;
48- } catch { /* a corrupt store starts fresh */ }
49- return [];
50-}
51-function save(): void {
52- localStorage.setItem(STORE_KEY, JSON.stringify({ voices }));
53-}
54-function voiceById(id: string): Voice | undefined {
55- return voices.find((v) => v.id === id);
56-}
57-function status(msg: string): void {
58- $('status').textContent = msg;
59-}
53+const iw = new IwEngine();
54+let bakeJob: { name: string; cancel: boolean } | null = null;
55+let bakeProg: HTMLProgressElement | null = null;
56+
57+function save(): void { saveVoices(voices); }
58+function voiceById(id: string): Voice | undefined { return voices.find((v) => v.id === id); }
59+function status(msg: string): void { $('status').textContent = msg; }
6060
6161 // ---- audition -------------------------------------------------------------
6262
63-function playSlice(clip: Clip, s0: number, s1: number): void {
63+function playSlice(samples: Float32Array, sr: number, s0: number, s1: number): void {
6464 const ctx = audio();
65- const n = Math.min(s1, s0 + Math.floor(0.5 * clip.sr)) - s0;
65+ const n = Math.min(s1, s0 + Math.floor(0.5 * sr)) - s0;
6666 if (n <= 0) return;
67- const buf = ctx.createBuffer(1, n, clip.sr);
68- buf.getChannelData(0).set(clip.samples.subarray(s0, s0 + n));
67+ const buf = ctx.createBuffer(1, n, sr);
68+ buf.getChannelData(0).set(samples.subarray(s0, s0 + n));
6969 const src = ctx.createBufferSource();
7070 src.buffer = buf;
7171 src.connect(ctx.destination);
@@ -149,94 +149,171 @@ async function stopRecording(): Promise<void> {
149149 renderVoiceList();
150150 syncTakeBar();
151151 if (target.kind === 'voice') {
152- pending = { clip, voiceId: target.voiceId, onsets: [], excluded: new Set() };
153- redetectPending();
154- $('capture').hidden = false;
152+ const voice = voiceById(target.voiceId);
153+ if (!voice) return;
154+ const sens = Number(($('capture-sens') as HTMLInputElement).value);
155+ const tclip: TrainClip = {
156+ id: uid(), sr: clip.sr, sens,
157+ onsets: detectOnsets(clip.samples, clip.sr, sens),
158+ excluded: [], hits: [],
159+ };
160+ recomputeHits(tclip, clip.samples);
161+ voice.clips.push(tclip);
162+ save();
163+ void putClipAudio(tclip.id, clip.samples, clip.sr).catch(() =>
164+ status('warning: clip audio could not be stored; this clip will not be re-editable after a reload'));
165+ openEditor(voice.id, tclip.id, { samples: clip.samples, sr: clip.sr });
166+ renderVoiceList();
155167 } else {
156168 take.clip = clip;
157169 processTake();
158170 }
159171 }
160172
161-// ---- capture review (voice training) ---------------------------------------
173+// ---- training clip editor ----------------------------------------------------
162174
163175 const waveView = new WaveView($('capture-wave') as HTMLCanvasElement);
164176
165-function redetectPending(): void {
166- if (!pending) return;
167- const sens = Number(($('capture-sens') as HTMLInputElement).value);
168- pending.onsets = detectOnsets(pending.clip.samples, pending.clip.sr, sens);
169- pending.excluded = new Set();
170- refreshCapture();
177+function editingClip(): TrainClip | null {
178+ if (!edit) return null;
179+ return voiceById(edit.voiceId)?.clips.find((c) => c.id === edit!.clipId) ?? null;
171180 }
172181
173-function refreshCapture(): void {
174- if (!pending) return;
175- const voice = voiceById(pending.voiceId)!;
176- const markers: Marker[] = pending.onsets.map((t, i) => ({ t, excluded: pending!.excluded.has(i) }));
177- waveView.set(pending.clip, markers, voice.color);
178- const n = pending.onsets.length - pending.excluded.size;
179- $('capture-title').textContent = `${pending.onsets.length} hit${pending.onsets.length === 1 ? '' : 's'} detected`;
180- ($('capture-commit') as HTMLButtonElement).textContent = `Add ${n} hit${n === 1 ? '' : 's'} to "${voice.name}"`;
181- ($('capture-commit') as HTMLButtonElement).disabled = n === 0;
182- ($('capture-exclude') as HTMLButtonElement).disabled = true;
182+function recomputeHits(clip: TrainClip, samples: Float32Array): void {
183+ const ex = new Set(clip.excluded);
184+ clip.hits = [];
185+ for (let i = 0; i < clip.onsets.length; i++) {
186+ if (ex.has(i)) continue;
187+ const s0 = Math.floor(clip.onsets[i] * clip.sr);
188+ const s1 = i + 1 < clip.onsets.length ? Math.floor(clip.onsets[i + 1] * clip.sr) : samples.length;
189+ const f = extractHit(samples, clip.sr, s0, s1);
190+ clip.hits.push({ vec: f.vec, peakDb: f.peakDb });
191+ }
183192 }
184193
185-waveView.onMarkerClick = (i) => {
186- if (!pending) return;
187- waveView.selected = i;
188- waveView.draw();
189- const s0 = Math.floor(pending.onsets[i] * pending.clip.sr);
190- const s1 = i + 1 < pending.onsets.length ? Math.floor(pending.onsets[i + 1] * pending.clip.sr) : pending.clip.samples.length;
191- playSlice(pending.clip, s0, s1);
194+function openEditor(voiceId: string, clipId: string, audioData?: { samples: Float32Array; sr: number }): void {
195+ const voice = voiceById(voiceId);
196+ const clip = voice?.clips.find((c) => c.id === clipId);
197+ if (!voice || !clip) return;
198+ const open = (samples: Float32Array, sr: number) => {
199+ edit = { voiceId, clipId, samples, sr };
200+ ($('capture-sens') as HTMLInputElement).value = String(clip.sens);
201+ $('capture').hidden = false;
202+ refreshEditor();
203+ };
204+ if (audioData) open(audioData.samples, audioData.sr);
205+ else {
206+ void getClipAudio(clipId).then((a) => {
207+ if (a) open(a.samples, a.sr);
208+ else status('this clip\'s audio is missing (stored before audio persistence, or cleared); its hits still classify');
209+ });
210+ }
211+}
212+
213+function refreshEditor(): void {
214+ const voice = edit && voiceById(edit.voiceId);
215+ const clip = editingClip();
216+ if (!edit || !voice || !clip) return;
217+ $('capture-title').textContent = `"${voice.name}"`;
218+ const sel = $('capture-clip') as HTMLSelectElement;
219+ sel.textContent = '';
220+ voice.clips.forEach((c, i) => {
221+ const o = document.createElement('option');
222+ o.value = c.id;
223+ o.textContent = `clip ${i + 1} · ${c.hits.length} hit${c.hits.length === 1 ? '' : 's'}`;
224+ sel.appendChild(o);
225+ });
226+ sel.value = clip.id;
227+ const markers: Marker[] = clip.onsets.map((t, i) => ({ t, excluded: clip.excluded.includes(i) }));
228+ waveView.set({ samples: edit.samples, sr: edit.sr }, markers, voice.color);
229+ syncEditorButtons();
230+}
231+
232+function syncEditorButtons(): void {
233+ const clip = editingClip();
234+ const n = waveView.selection.size;
192235 const btn = $('capture-exclude') as HTMLButtonElement;
193- btn.disabled = false;
194- btn.textContent = pending.excluded.has(i) ? 'Include' : 'Exclude';
236+ btn.disabled = n === 0;
237+ if (clip && n > 0) {
238+ const allExcluded = [...waveView.selection].every((i) => clip.excluded.includes(i));
239+ btn.textContent = allExcluded ? `Include (${n})` : `Exclude (${n})`;
240+ } else btn.textContent = 'Exclude';
241+ $('capture-count').textContent = clip
242+ ? `${clip.hits.length} of ${clip.onsets.length} hits included${n ? ` · ${n} selected` : ''}`
243+ : '';
244+}
245+
246+function toggleExcluded(): void {
247+ const clip = editingClip();
248+ if (!edit || !clip || waveView.selection.size === 0) return;
249+ const sel = [...waveView.selection];
250+ const allExcluded = sel.every((i) => clip.excluded.includes(i));
251+ const ex = new Set(clip.excluded);
252+ for (const i of sel) {
253+ if (allExcluded) ex.delete(i);
254+ else ex.add(i);
255+ }
256+ clip.excluded = [...ex].sort((a, b) => a - b);
257+ recomputeHits(clip, edit.samples);
258+ save();
259+ waveView.updateMarkers(clip.onsets.map((t, i) => ({ t, excluded: ex.has(i) })));
260+ syncEditorButtons();
261+ renderVoiceList();
262+}
263+
264+waveView.onAudition = (i) => {
265+ const clip = editingClip();
266+ if (!edit || !clip) return;
267+ const s0 = Math.floor(clip.onsets[i] * edit.sr);
268+ const s1 = i + 1 < clip.onsets.length ? Math.floor(clip.onsets[i + 1] * edit.sr) : edit.samples.length;
269+ playSlice(edit.samples, edit.sr, s0, s1);
195270 };
271+waveView.onSelectionChange = syncEditorButtons;
196272
197-$('capture-exclude').addEventListener('click', () => {
198- const i = waveView.selected;
199- if (!pending || i < 0) return;
200- if (pending.excluded.has(i)) pending.excluded.delete(i);
201- else pending.excluded.add(i);
202- const sel = i;
203- refreshCapture();
204- waveView.selected = sel;
205- waveView.draw();
206- const btn = $('capture-exclude') as HTMLButtonElement;
207- btn.disabled = false;
208- btn.textContent = pending.excluded.has(i) ? 'Include' : 'Exclude';
273+$('capture-exclude').addEventListener('click', toggleExcluded);
274+
275+$('capture-sens').addEventListener('input', () => {
276+ const clip = editingClip();
277+ if (!edit || !clip) return;
278+ clip.sens = Number(($('capture-sens') as HTMLInputElement).value);
279+ clip.onsets = detectOnsets(edit.samples, edit.sr, clip.sens);
280+ clip.excluded = [];
281+ recomputeHits(clip, edit.samples);
282+ save();
283+ refreshEditor();
284+ renderVoiceList();
209285 });
210286
211-$('capture-sens').addEventListener('input', redetectPending);
212-
213-$('capture-commit').addEventListener('click', () => {
214- if (!pending) return;
215- const voice = voiceById(pending.voiceId)!;
216- const { samples, sr } = pending.clip;
217- let added = 0;
218- for (let i = 0; i < pending.onsets.length; i++) {
219- if (pending.excluded.has(i)) continue;
220- const s0 = Math.floor(pending.onsets[i] * sr);
221- const s1 = i + 1 < pending.onsets.length ? Math.floor(pending.onsets[i + 1] * sr) : samples.length;
222- const f = extractHit(samples, sr, s0, s1);
223- voice.train.push({ vec: f.vec, peakDb: f.peakDb });
224- added++;
225- }
287+$('capture-clip').addEventListener('change', () => {
288+ if (!edit) return;
289+ openEditor(edit.voiceId, ($('capture-clip') as HTMLSelectElement).value);
290+});
291+
292+$('capture-delete').addEventListener('click', () => {
293+ const voice = edit && voiceById(edit.voiceId);
294+ const clip = editingClip();
295+ if (!edit || !voice || !clip) return;
296+ if (!confirm(`Delete this clip and its ${clip.hits.length} trained hits?`)) return;
297+ void deleteClipAudio(clip.id);
298+ voice.clips = voice.clips.filter((c) => c.id !== clip.id);
226299 save();
227- pending = null;
228- $('capture').hidden = true;
229300 renderVoiceList();
230- status(`added ${added} hit${added === 1 ? '' : 's'} to "${voice.name}" (${voice.train.length} total)`);
301+ if (voice.clips.length) openEditor(voice.id, voice.clips[voice.clips.length - 1].id);
302+ else closeEditor();
231303 });
232304
233-$('capture-discard').addEventListener('click', () => {
234- pending = null;
305+function closeEditor(): void {
306+ edit = null;
235307 $('capture').hidden = true;
236-});
308+}
309+$('capture-done').addEventListener('click', closeEditor);
237310
238311 // ---- voices ---------------------------------------------------------------
239312
313+function bindingValue(v: Voice): string {
314+ return v.binding.kind === 'synth' ? `synth:${v.binding.preset}` : `iw:${v.binding.instrument}`;
315+}
316+
240317 function renderVoiceList(): void {
241318 const list = $('voice-list');
242319 list.textContent = '';
@@ -257,20 +334,13 @@ function renderVoiceList(): void {
257334 name.value = voice.name;
258335 save();
259336 refreshTimeline();
337+ if (edit?.voiceId === voice.id) refreshEditor();
260338 });
261339
340+ const n = trainPool(voice).length;
262341 const count = document.createElement('span');
263342 count.className = 'count';
264- count.textContent = `${voice.train.length} trained hit${voice.train.length === 1 ? '' : 's'}`;
265-
266- const preset = document.createElement('select');
267- for (const p of PRESETS) {
268- const o = document.createElement('option');
269- o.value = p; o.textContent = p;
270- preset.appendChild(o);
271- }
272- preset.value = voice.preset;
273- preset.addEventListener('change', () => { voice.preset = preset.value as SynthPreset; save(); });
343+ count.textContent = `${voice.clips.length} clip${voice.clips.length === 1 ? '' : 's'} · ${n} hit${n === 1 ? '' : 's'}`;
274344
275345 const rec = document.createElement('button');
276346 const recordingThis = recTarget?.kind === 'voice' && recTarget.voiceId === voice.id;
@@ -282,13 +352,64 @@ function renderVoiceList(): void {
282352 else void startRecording({ kind: 'voice', voiceId: voice.id });
283353 });
284354
355+ const editBtn = document.createElement('button');
356+ editBtn.textContent = 'Edit';
357+ editBtn.title = 'reopen this voice\'s training clips';
358+ editBtn.disabled = voice.clips.length === 0;
359+ editBtn.addEventListener('click', () => {
360+ openEditor(voice.id, voice.clips[voice.clips.length - 1].id);
361+ });
362+
363+ const binding = document.createElement('select');
364+ const gSynth = document.createElement('optgroup');
365+ gSynth.label = 'synth';
366+ for (const p of PRESETS) {
367+ const o = document.createElement('option');
368+ o.value = `synth:${p}`; o.textContent = p;
369+ gSynth.appendChild(o);
370+ }
371+ const gIw = document.createElement('optgroup');
372+ gIw.label = 'instrawave';
373+ for (const nm of iw.names()) {
374+ const o = document.createElement('option');
375+ o.value = `iw:${nm}`; o.textContent = nm;
376+ gIw.appendChild(o);
377+ }
378+ binding.append(gSynth, gIw);
379+ binding.value = bindingValue(voice);
380+ binding.addEventListener('change', () => {
381+ const [kind, which] = binding.value.split(':');
382+ if (kind === 'synth') voice.binding = { kind: 'synth', preset: which as SynthPreset };
383+ else {
384+ const old = voice.binding;
385+ const set = old.kind === 'iw' && old.instrument === which ? old.set : {};
386+ voice.binding = { kind: 'iw', instrument: which, set };
387+ }
388+ save();
389+ renderVoiceList();
390+ updateBakeBar();
391+ });
392+
393+ const tune = document.createElement('button');
394+ tune.textContent = expandedTuning.has(voice.id) ? 'Tune ▾' : 'Tune ▸';
395+ tune.disabled = voice.binding.kind !== 'iw';
396+ tune.addEventListener('click', () => {
397+ if (expandedTuning.has(voice.id)) expandedTuning.delete(voice.id);
398+ else expandedTuning.add(voice.id);
399+ renderVoiceList();
400+ });
401+
285402 const clear = document.createElement('button');
286403 clear.textContent = 'Clear';
287- clear.title = 'forget this voice\'s trained hits';
288- clear.disabled = voice.train.length === 0;
404+ clear.title = 'forget this voice\'s training';
405+ clear.disabled = n === 0;
289406 clear.addEventListener('click', () => {
290- voice.train = [];
407+ if (!confirm(`Forget all ${n} trained hits of "${voice.name}"?`)) return;
408+ for (const c of voice.clips) void deleteClipAudio(c.id);
409+ voice.clips = [];
410+ voice.loose = [];
291411 save();
412+ if (edit?.voiceId === voice.id) closeEditor();
292413 renderVoiceList();
293414 });
294415
@@ -296,21 +417,58 @@ function renderVoiceList(): void {
296417 del.textContent = '✕';
297418 del.title = 'delete voice';
298419 del.addEventListener('click', () => {
299- if (!confirm(`Delete voice "${voice.name}"? Its trained hits and its timeline events go with it.`)) return;
420+ if (!confirm(`Delete voice "${voice.name}"? Its training and its timeline events go with it.`)) return;
421+ for (const c of voice.clips) void deleteClipAudio(c.id);
300422 voices = voices.filter((v) => v.id !== voice.id);
301423 take.hits = take.hits.filter((h) => h.voiceId !== voice.id);
302424 save();
425+ if (edit?.voiceId === voice.id) closeEditor();
303426 renderVoiceList();
304427 refreshTimeline();
428+ updateBakeBar();
305429 });
306430
307431 const spacer = document.createElement('span');
308432 spacer.className = 'spacer';
309- row.append(swatch, name, count, rec, clear, spacer, preset, del);
433+ row.append(swatch, name, count, rec, editBtn, clear, spacer, binding, tune, del);
310434 list.appendChild(row);
435+
436+ if (voice.binding.kind === 'iw' && expandedTuning.has(voice.id)) {
437+ list.appendChild(tuningPanel(voice, voice.binding.instrument, voice.binding.set));
438+ }
311439 }
312440 }
313441
442+function tuningPanel(voice: Voice, instrument: string, set: Record<string, number>): HTMLElement {
443+ const panel = document.createElement('div');
444+ panel.className = 'tuning-panel';
445+ for (const p of iw.tuning(instrument)) {
446+ const div = document.createElement('div');
447+ div.className = 'param';
448+ const label = document.createElement('label');
449+ const nameSpan = document.createElement('span');
450+ nameSpan.textContent = `${p.label} (${p.unit})`;
451+ const valSpan = document.createElement('span');
452+ const input = document.createElement('input');
453+ input.type = 'range';
454+ input.min = String(p.min);
455+ input.max = String(p.max);
456+ input.step = String(p.step ?? (p.max - p.min) / 200);
457+ input.value = String(set[p.key] ?? p.value);
458+ const show = () => { valSpan.textContent = Number(input.value).toPrecision(4); };
459+ show();
460+ input.addEventListener('input', () => {
461+ set[p.key] = Number(input.value);
462+ show();
463+ save();
464+ });
465+ label.append(nameSpan, valSpan);
466+ div.append(label, input);
467+ panel.appendChild(div);
468+ }
469+ return panel;
470+}
471+
314472 $('add-voice').addEventListener('click', () => {
315473 const input = $('voice-name') as HTMLInputElement;
316474 const name = input.value.trim() || `voice ${voices.length + 1}`;
@@ -318,8 +476,9 @@ $('add-voice').addEventListener('click', () => {
318476 id: uid(),
319477 name,
320478 color: PALETTE[voices.length % PALETTE.length],
321- preset: PRESETS[voices.length % PRESETS.length],
322- train: [],
479+ binding: { kind: 'synth', preset: PRESETS[voices.length % PRESETS.length] },
480+ clips: [],
481+ loose: [],
323482 });
324483 input.value = '';
325484 save();
@@ -329,17 +488,80 @@ $('add-voice').addEventListener('click', () => {
329488
330489 $('reset-all').addEventListener('click', () => {
331490 if (!confirm('Forget all voices, training, and the current take?')) return;
332- localStorage.removeItem(STORE_KEY);
491+ clearStore();
492+ void clearClipAudio();
333493 voices = [];
334494 take = { clip: null, hits: [] };
335- pending = null;
336- $('capture').hidden = true;
495+ closeEditor();
337496 renderVoiceList();
338497 refreshTimeline();
339498 syncTakeBar();
499+ updateBakeBar();
340500 status('');
341501 });
342502
503+// ---- bakes ------------------------------------------------------------------
504+
505+function neededInstruments(): string[] {
506+ return [...new Set(voices.filter((v) => v.binding.kind === 'iw')
507+ .map((v) => (v.binding as { instrument: string }).instrument))];
508+}
509+
510+function updateBakeBar(): void {
511+ const bar = $('bakebar');
512+ const needed = neededInstruments();
513+ bar.hidden = needed.length === 0;
514+ bar.textContent = '';
515+ bakeProg = null;
516+ for (const name of needed) {
517+ const label = document.createElement('span');
518+ label.className = 'hint';
519+ label.textContent = `${name} bake:`;
520+ bar.appendChild(label);
521+ if (iw.hasBake(name)) {
522+ const ok = document.createElement('span');
523+ ok.textContent = 'ready';
524+ ok.style.color = '#15803d';
525+ bar.appendChild(ok);
526+ } else if (bakeJob?.name === name) {
527+ const prog = document.createElement('progress');
528+ prog.max = 1;
529+ prog.value = 0;
530+ bakeProg = prog;
531+ const cancel = document.createElement('button');
532+ cancel.textContent = 'Cancel';
533+ cancel.addEventListener('click', () => { if (bakeJob) bakeJob.cancel = true; });
534+ bar.append(prog, cancel);
535+ } else {
536+ const btn = document.createElement('button');
537+ btn.textContent = 'Precompute';
538+ btn.disabled = bakeJob !== null;
539+ btn.addEventListener('click', () => void startBake(name));
540+ bar.appendChild(btn);
541+ }
542+ }
543+}
544+
545+async function startBake(name: string): Promise<void> {
546+ if (bakeJob) return;
547+ bakeJob = { name, cancel: false };
548+ updateBakeBar();
549+ try {
550+ await iw.precompute(
551+ name,
552+ (done, total) => { if (bakeProg) bakeProg.value = done / total; },
553+ () => bakeJob?.cancel ?? true,
554+ );
555+ status(`${name} bake ready`);
556+ } catch (e) {
557+ status(e instanceof PhaseCancelled ? `${name} precompute cancelled`
558+ : `${name} precompute failed: ${e instanceof Error ? e.message : e}`);
559+ } finally {
560+ bakeJob = null;
561+ updateBakeBar();
562+ }
563+}
564+
343565 // ---- take -----------------------------------------------------------------
344566
345567 const timeline = new Timeline($('timeline') as HTMLCanvasElement);
@@ -350,10 +572,10 @@ function processTake(): void {
350572 const { samples, sr } = take.clip;
351573 const sens = Number(($('take-sens') as HTMLInputElement).value);
352574 const onsets = detectOnsets(samples, sr, sens);
353- const trained = voices.filter((v) => v.train.length > 0);
575+ const trained = voices.filter((v) => trainPool(v).length > 0);
354576 const trainSet: TrainingSet = {
355- voiceIds: trained.flatMap((v) => v.train.map(() => v.id)),
356- vecs: trained.flatMap((v) => v.train.map((h) => h.vec)),
577+ voiceIds: trained.flatMap((v) => trainPool(v).map(() => v.id)),
578+ vecs: trained.flatMap((v) => trainPool(v).map((h) => h.vec)),
357579 };
358580
359581 const hits: TakeHit[] = [];
@@ -419,14 +641,14 @@ timeline.onSelect = (id) => {
419641 syncHitButtons();
420642 const hit = take.hits.find((h) => h.id === id);
421643 if (!hit) return;
422- if (hit.s0 >= 0 && take.clip) playSlice(take.clip, hit.s0, hit.s1);
423- else void auditionSynth(hit);
644+ const voice = voiceById(hit.voiceId);
645+ status(`hit at ${hit.t.toFixed(3)} s · "${voice?.name ?? '?'}" · strength ${hit.gain.toFixed(2)}`);
646+ if (hit.s0 >= 0 && take.clip) playSlice(take.clip.samples, take.clip.sr, hit.s0, hit.s1);
647+ else if (voice?.binding.kind === 'synth') void auditionSynth(hit, voice.binding.preset);
424648 };
425649
426-async function auditionSynth(hit: TakeHit): Promise<void> {
427- const voice = voiceById(hit.voiceId);
428- if (!voice) return;
429- const buf = await renderSynth([{ t: 0, voiceId: hit.voiceId, gain: hit.gain }], () => voice.preset, 1);
650+async function auditionSynth(hit: TakeHit, preset: SynthPreset): Promise<void> {
651+ const buf = await renderSynth([{ t: 0, voiceId: hit.voiceId, gain: hit.gain }], () => preset, 1);
430652 void playBuffer(buf);
431653 }
432654
@@ -444,9 +666,15 @@ timeline.onAdd = (t, voiceId) => {
444666
445667 $('hit-delete').addEventListener('click', deleteSelected);
446668 document.addEventListener('keydown', (e) => {
447- if (e.key !== 'Delete' && e.key !== 'Backspace') return;
448669 if (document.activeElement && ['INPUT', 'SELECT', 'TEXTAREA'].includes(document.activeElement.tagName)) return;
449- deleteSelected();
670+ if (!$('capture').hidden && edit) {
671+ if (e.key === 'x' || e.key === 'X' || e.key === 'Delete' || e.key === 'Backspace') {
672+ e.preventDefault();
673+ toggleExcluded();
674+ }
675+ return;
676+ }
677+ if (e.key === 'Delete' || e.key === 'Backspace') deleteSelected();
450678 });
451679 function deleteSelected(): void {
452680 if (!timeline.selected) return;
@@ -460,19 +688,92 @@ $('hit-learn').addEventListener('click', () => {
460688 if (!hit || !hit.vec) return;
461689 const voice = voiceById(hit.voiceId);
462690 if (!voice) return;
463- voice.train.push({ vec: hit.vec, peakDb: hit.peakDb });
691+ voice.loose.push({ vec: hit.vec, peakDb: hit.peakDb });
464692 save();
465693 renderVoiceList();
466- status(`learned this hit as "${voice.name}" (${voice.train.length} trained)`);
694+ status(`learned this hit as "${voice.name}" (${trainPool(voice).length} trained)`);
467695 });
468696
697+// ---- the mix ----------------------------------------------------------------
698+
699+async function renderMix(): Promise<AudioBuffer> {
700+ const seconds = Math.max(...take.hits.map((h) => h.t)) + 1.5;
701+ const sig = JSON.stringify({
702+ hits: take.hits.map((h) => [h.t, h.voiceId, h.gain]),
703+ bindings: voices.map((v) => [v.id, v.binding]),
704+ });
705+ if (renderCache?.sig === sig) return renderCache.buf;
706+
707+ const synthEvents: NoteEvent[] = [];
708+ const iwEvents = new Map<string, StimulusEvent[]>();
709+ for (const h of take.hits) {
710+ const v = voiceById(h.voiceId);
711+ if (!v) continue;
712+ if (v.binding.kind === 'synth') {
713+ synthEvents.push({ t: h.t, voiceId: h.voiceId, gain: h.gain });
714+ } else {
715+ const b = v.binding;
716+ const allowed = new Set(iw.tuning(b.instrument).map((p) => p.key));
717+ const set: Record<string, number> = {};
718+ for (const [k, val] of Object.entries(b.set)) if (allowed.has(k)) set[k] = val;
719+ const strength = iw.strengthParam(b.instrument);
720+ set[strength] = (b.set[strength] ?? iw.defaultValue(b.instrument, strength)) * h.gain;
721+ let list = iwEvents.get(b.instrument);
722+ if (!list) iwEvents.set(b.instrument, (list = []));
723+ list.push({ t: h.t, set });
724+ }
725+ }
726+ const missing = [...iwEvents.keys()].filter((n) => !iw.hasBake(n));
727+ if (missing.length) {
728+ throw new Error(`precompute the ${missing.join(' and ')} bake${missing.length > 1 ? 's' : ''} first`);
729+ }
730+
731+ const out = new Float32Array(Math.ceil(seconds * MIX_SR));
732+ if (synthEvents.length) {
733+ const buf = await renderSynth(
734+ synthEvents,
735+ (id) => {
736+ const v = voiceById(id);
737+ return v?.binding.kind === 'synth' ? v.binding.preset : 'kick';
738+ },
739+ seconds,
740+ );
741+ mixInto(out, buf.getChannelData(0));
742+ }
743+ for (const [name, evs] of iwEvents) {
744+ evs.sort((a, b) => a.t - b.t);
745+ const { trace, dt } = await iw.render(name, seconds, evs, (done, total) =>
746+ status(`rendering ${name} (${evs.length} events): ${Math.round((100 * done) / total)}%`));
747+ // Physical pressure amplitudes carry no absolute scale worth keeping
748+ // across engines; each instrument's trace is levelled to sit alongside
749+ // the synth part, while relative volumes WITHIN it stay physical.
750+ const res = resampleLinear(trace, 1 / dt, MIX_SR);
751+ const p = peak(res);
752+ if (p > 0) mixInto(out, res, 0.8 / p);
753+ }
754+ const pAll = peak(out);
755+ if (pAll > 0.98) for (let i = 0; i < out.length; i++) out[i] *= 0.95 / pAll;
756+ const buf = audio().createBuffer(1, out.length, MIX_SR);
757+ buf.getChannelData(0).set(out);
758+ renderCache = { sig, buf };
759+ return buf;
760+}
761+
469762 $('take-play').addEventListener('click', async () => {
470763 if (playing) { stopPlayback(); return; }
471764 if (take.hits.length === 0) return;
472- const events: NoteEvent[] = take.hits.map((h) => ({ t: h.t, voiceId: h.voiceId, gain: h.gain }));
473- const seconds = Math.max(...events.map((e) => e.t)) + 1.5;
474- const buf = await renderSynth(events, (id) => voiceById(id)?.preset ?? 'kick', seconds);
475- void playBuffer(buf, (t) => timeline.playhead(t));
765+ const btn = $('take-play') as HTMLButtonElement;
766+ btn.disabled = true;
767+ try {
768+ const buf = await renderMix();
769+ status('');
770+ void playBuffer(buf, (t) => timeline.playhead(t));
771+ } catch (e) {
772+ status(e instanceof Error ? e.message : String(e));
773+ } finally {
774+ btn.disabled = false;
775+ syncPlayButton();
776+ }
476777 });
477778
478779 // ---- boot -----------------------------------------------------------------
@@ -480,6 +781,7 @@ $('take-play').addEventListener('click', async () => {
480781 renderVoiceList();
481782 refreshTimeline();
482783 syncTakeBar();
784+updateBakeBar();
483785 if (voices.length === 0) {
484786 status('start by adding a voice, then record yourself making its sound a handful of times');
485787 }
src/render/instrawave.tsadded+96−0View file
@@ -0,0 +1,96 @@
1+/**
2+ * The instrawave side of the renderer seam. The three bundled instruments
3+ * are held at their default geometry and accuracy, so each has exactly one
4+ * bake, computed on demand and kept for the session (the drum's is a
5+ * couple of hundred MB; persisting it is future work). All per-voice
6+ * character travels as tuning, which is free: every voice bound to the
7+ * same instrument shares its bake, and one render call carries all their
8+ * events, each with the voice's tuning overrides and its strength scaling
9+ * the instrument's excitation amplitude (`vel`, or `amp` where there is no
10+ * strike).
11+ */
12+import { requestDevice } from 'instrawave/src/host/device.ts';
13+import { InstrumentSession } from 'instrawave/src/host/session.ts';
14+import {
15+ parseInstrument,
16+ paramValues,
17+ type InstrumentFile,
18+ type ParamSpec,
19+ type StimulusEvent,
20+} from 'instrawave/src/format.ts';
21+import type { Bake } from 'instrawave/src/host/bake.ts';
22+import drumJson from 'instrawave/instruments/drum.instrument.json';
23+import dulcimerJson from 'instrawave/instruments/dulcimer.instrument.json';
24+import chimeJson from 'instrawave/instruments/chime.instrument.json';
25+
26+export { PhaseCancelled } from 'instrawave/src/host/session.ts';
27+export type { StimulusEvent };
28+
29+const files: InstrumentFile[] = [drumJson, dulcimerJson, chimeJson].map((j) =>
30+ parseInstrument(JSON.stringify(j)),
31+);
32+
33+export type Progress = (done: number, total: number) => void;
34+
35+export class IwEngine {
36+ private device: GPUDevice | null = null;
37+ private bakes = new Map<string, Bake>();
38+ private insts = new Map(files.map((f) => [f.name, f]));
39+
40+ names(): string[] {
41+ return [...this.insts.keys()];
42+ }
43+
44+ private inst(name: string): InstrumentFile {
45+ const f = this.insts.get(name);
46+ if (!f) throw new Error(`unknown instrument '${name}'`);
47+ return f;
48+ }
49+
50+ /** The tuning parameters a binding may override: tuning-class params
51+ * that are also event parameters, so they can vary per hit. */
52+ tuning(name: string): ParamSpec[] {
53+ const f = this.inst(name);
54+ return f.params.filter((p) => p.class === 'tuning' && f.render.eventParams.includes(p.key));
55+ }
56+
57+ /** The parameter a hit's strength scales: the strike speed where there
58+ * is one, the pluck amplitude otherwise. Linear equations, so it is a
59+ * pure scale. */
60+ strengthParam(name: string): string {
61+ const ep = this.inst(name).render.eventParams;
62+ return ep.includes('vel') ? 'vel' : 'amp';
63+ }
64+
65+ defaultValue(name: string, key: string): number {
66+ const p = this.inst(name).params.find((q) => q.key === key);
67+ if (!p) throw new Error(`instrument '${name}' has no parameter '${key}'`);
68+ return p.value;
69+ }
70+
71+ hasBake(name: string): boolean {
72+ return this.bakes.has(name);
73+ }
74+
75+ async precompute(name: string, onProgress?: Progress, cancelled?: () => boolean): Promise<void> {
76+ const inst = this.inst(name);
77+ this.device ??= await requestDevice();
78+ const session = new InstrumentSession(this.device, inst, paramValues(inst));
79+ this.bakes.set(name, await session.precompute(onProgress, cancelled));
80+ }
81+
82+ async render(
83+ name: string,
84+ seconds: number,
85+ events: StimulusEvent[],
86+ onProgress?: Progress,
87+ ): Promise<{ trace: Float32Array; dt: number }> {
88+ const inst = this.inst(name);
89+ const bake = this.bakes.get(name);
90+ if (!bake) throw new Error(`no bake for '${name}'; precompute first`);
91+ if (!this.device) throw new Error('no GPU device');
92+ const session = new InstrumentSession(this.device, inst, paramValues(inst));
93+ const { trace, dt } = await session.render(bake, seconds, events, [], onProgress);
94+ return { trace, dt };
95+ }
96+}
src/render/mix.tsadded+33−0View file
@@ -0,0 +1,33 @@
1+/**
2+ * Small pure helpers for combining traces from different renderers at
3+ * different rates. Resampling is linear interpolation, which is adequate
4+ * here because the instrawave traces arrive oversampled (the solver rate,
5+ * around 114 kHz) and the content sits far below either rate.
6+ */
7+export function resampleLinear(x: Float32Array, srFrom: number, srTo: number): Float32Array {
8+ if (x.length === 0) return new Float32Array(0);
9+ const n = Math.max(1, Math.round((x.length * srTo) / srFrom));
10+ const out = new Float32Array(n);
11+ for (let i = 0; i < n; i++) {
12+ const s = (i * srFrom) / srTo;
13+ const k = Math.min(x.length - 2, Math.floor(s));
14+ const f = Math.min(1, Math.max(0, s - k));
15+ out[i] = (1 - f) * x[k] + f * x[Math.min(x.length - 1, k + 1)];
16+ }
17+ return out;
18+}
19+
20+export function peak(x: Float32Array): number {
21+ let p = 0;
22+ for (let i = 0; i < x.length; i++) {
23+ const a = Math.abs(x[i]);
24+ if (a > p) p = a;
25+ }
26+ return p;
27+}
28+
29+/** dst += src * gain, clipped to dst's length. */
30+export function mixInto(dst: Float32Array, src: Float32Array, gain = 1): void {
31+ const n = Math.min(dst.length, src.length);
32+ for (let i = 0; i < n; i++) dst[i] += src[i] * gain;
33+}
src/render/types.tsmodified+7−7View file
@@ -1,10 +1,10 @@
11 /**
22 * The seam between the analysis half and the sound half. Everything above
33 * this interface deals in NoteEvents (a time, a voice, a strength);
4- * everything below turns them into audio. The synth renderer is the first
5- * implementation; the instrawave renderer replaces a binding's preset with
6- * an instrument file plus parameter values, groups events by bake key so
7- * voices sharing a bake render in one pass, and mixes the traces.
4+ * everything below turns them into audio. A voice's binding names either a
5+ * placeholder synth preset or an instrawave instrument with tuning
6+ * overrides; the mixer in main.ts sends each event down the right path and
7+ * touches the GPU only when at least one bound instrument is instrawave.
88 */
99 export interface NoteEvent {
1010 t: number; // seconds from the start of the take
@@ -14,9 +14,9 @@ export interface NoteEvent {
1414
1515 export type SynthPreset = 'kick' | 'snare' | 'hat' | 'tom';
1616
17-export interface Renderer {
18- render(events: NoteEvent[], preset: (voiceId: string) => SynthPreset, seconds: number): Promise<AudioBuffer>;
19-}
17+export type Binding =
18+ | { kind: 'synth'; preset: SynthPreset }
19+ | { kind: 'iw'; instrument: string; set: Record<string, number> };
2020
2121 /** Map a hit's peak level to a note strength. Levels are taken relative to
2222 * the loudest hit in the take and compressed (0.5 exponent over a 30 dB
src/store.tsadded+101−0View file
@@ -0,0 +1,101 @@
1+/**
2+ * Persistence. Voice metadata, training-clip descriptions, and cached
3+ * feature vectors live in localStorage; the clips' raw audio lives in
4+ * IndexedDB keyed by clip id, so a training can be reopened, re-detected,
5+ * and re-labelled later. The two stores can drift only in one harmless
6+ * direction: a clip whose audio is missing simply cannot be re-edited,
7+ * while its cached features keep classifying.
8+ */
9+import type { Binding } from './render/types.ts';
10+
11+export interface TrainHit { vec: number[]; peakDb: number; }
12+export interface TrainClip {
13+ id: string;
14+ sr: number;
15+ sens: number; // the sensitivity the onsets were detected with
16+ onsets: number[]; // seconds
17+ excluded: number[]; // indices into onsets
18+ hits: TrainHit[]; // features of the included onsets, cached
19+}
20+export interface Voice {
21+ id: string;
22+ name: string;
23+ color: string;
24+ binding: Binding;
25+ clips: TrainClip[];
26+ loose: TrainHit[]; // hits learned from takes; they belong to no clip
27+}
28+
29+/** The pooled training set a voice contributes to classification. */
30+export function trainPool(v: Voice): TrainHit[] {
31+ return [...v.clips.flatMap((c) => c.hits), ...v.loose];
32+}
33+
34+const KEY = 'instrawave-compose/v2';
35+const KEY_V1 = 'instrawave-compose/v1';
36+
37+export function loadVoices(): Voice[] {
38+ try {
39+ const raw = localStorage.getItem(KEY);
40+ if (raw) return (JSON.parse(raw) as { voices: Voice[] }).voices;
41+ const old = localStorage.getItem(KEY_V1);
42+ if (old) {
43+ // v1 stored a flat training list and a synth preset per voice.
44+ const v1 = JSON.parse(old) as { voices: { id: string; name: string; color: string; preset: string; train: TrainHit[] }[] };
45+ const voices: Voice[] = v1.voices.map((v) => ({
46+ id: v.id, name: v.name, color: v.color,
47+ binding: { kind: 'synth', preset: v.preset as ('kick' | 'snare' | 'hat' | 'tom') },
48+ clips: [], loose: v.train,
49+ }));
50+ localStorage.setItem(KEY, JSON.stringify({ voices }));
51+ localStorage.removeItem(KEY_V1);
52+ return voices;
53+ }
54+ } catch { /* a corrupt store starts fresh */ }
55+ return [];
56+}
57+
58+export function saveVoices(voices: Voice[]): void {
59+ localStorage.setItem(KEY, JSON.stringify({ voices }));
60+}
61+
62+export function clearStore(): void {
63+ localStorage.removeItem(KEY);
64+}
65+
66+// ---- clip audio in IndexedDB ------------------------------------------------
67+
68+let dbPromise: Promise<IDBDatabase> | null = null;
69+function db(): Promise<IDBDatabase> {
70+ return (dbPromise ??= new Promise((resolve, reject) => {
71+ const req = indexedDB.open('instrawave-compose', 1);
72+ req.onupgradeneeded = () => req.result.createObjectStore('clips');
73+ req.onsuccess = () => resolve(req.result);
74+ req.onerror = () => reject(req.error);
75+ }));
76+}
77+
78+function op<T>(mode: IDBTransactionMode, fn: (store: IDBObjectStore) => IDBRequest<T>): Promise<T> {
79+ return db().then((d) => new Promise<T>((resolve, reject) => {
80+ const req = fn(d.transaction('clips', mode).objectStore('clips'));
81+ req.onsuccess = () => resolve(req.result);
82+ req.onerror = () => reject(req.error);
83+ }));
84+}
85+
86+export function putClipAudio(id: string, samples: Float32Array, sr: number): Promise<unknown> {
87+ return op('readwrite', (s) => s.put({ samples, sr }, id));
88+}
89+
90+export function getClipAudio(id: string): Promise<{ samples: Float32Array; sr: number } | null> {
91+ return op<{ samples: Float32Array; sr: number } | undefined>('readonly', (s) => s.get(id))
92+ .then((r) => r ?? null);
93+}
94+
95+export function deleteClipAudio(id: string): Promise<unknown> {
96+ return op('readwrite', (s) => s.delete(id));
97+}
98+
99+export function clearClipAudio(): Promise<unknown> {
100+ return op('readwrite', (s) => s.clear());
101+}
src/ui/timeline.tsmodified+21−0View file
@@ -203,6 +203,27 @@ export class Timeline {
203203 g.globalAlpha = 1;
204204 }
205205
206+ // Pulses: each hit flashes as the playhead crosses it, ring radius
207+ // proportional to its strength, so the dynamics are visible in motion.
208+ if (this.playT !== null) {
209+ for (const ev of this.events) {
210+ const p = (this.playT - ev.t) / 0.3;
211+ if (p < 0 || p > 1) continue;
212+ const li = this.lanes.findIndex((l) => l.id === ev.voiceId);
213+ if (li < 0) continue;
214+ const x = this.xOf(ev.t, w);
215+ const cy = STRIP_H + li * LANE_H + LANE_H / 2;
216+ const r = (4 + 16 * ev.gain) * (0.4 + 0.6 * p);
217+ g.strokeStyle = this.lanes[li].color;
218+ g.globalAlpha = (1 - p) * 0.9;
219+ g.lineWidth = 2;
220+ g.beginPath();
221+ g.arc(x, cy, r, 0, 2 * Math.PI);
222+ g.stroke();
223+ g.globalAlpha = 1;
224+ }
225+ }
226+
206227 // Playhead.
207228 if (this.playT !== null) {
208229 const x = this.xOf(this.playT, w);
src/ui/waveform.tsmodified+103−25View file
@@ -1,9 +1,10 @@
11 /**
2- * The capture review: one recorded clip drawn as a waveform with a marker
3- * per detected onset. Markers audition on click and can be excluded (drawn
4- * hollow) before the hits are committed to a voice's training set, so a
5- * cough or a spurious detection is visible and removable rather than
6- * silently poisoning the label.
2+ * The training-clip editor's view: one clip as a waveform with a marker
3+ * per detected onset. Click selects one marker and auditions it;
4+ * shift- or ctrl-click toggles a marker in the selection; dragging over
5+ * empty space sweeps a rubber-band selection. Excluded markers draw
6+ * hollow. Exclusion itself is the owner's business (a button or the X
7+ * key); this view only manages what is selected.
78 */
89 import type { Clip } from '../audio/recorder.ts';
910
@@ -20,24 +21,16 @@ export class WaveView {
2021 private clip: Clip | null = null;
2122 private markers: Marker[] = [];
2223 private color = '#e4573d';
23- selected = -1;
24- onMarkerClick: ((index: number) => void) | null = null;
24+ private band: { x0: number; x1: number; base: Set<number>; moved: boolean; additive: boolean } | null = null;
25+ selection = new Set<number>();
26+ onAudition: ((index: number) => void) | null = null;
27+ onSelectionChange: (() => void) | null = null;
2528
2629 constructor(canvas: HTMLCanvasElement) {
2730 this.canvas = canvas;
28- canvas.addEventListener('pointerdown', (e) => {
29- if (!this.clip) return;
30- const rect = canvas.getBoundingClientRect();
31- const x = e.clientX - rect.left;
32- const dur = this.clip.samples.length / this.clip.sr;
33- let best = -1, bestDx = 8;
34- for (let i = 0; i < this.markers.length; i++) {
35- const mx = (this.markers[i].t / dur) * rect.width;
36- const dx = Math.abs(mx - x);
37- if (dx < bestDx) { best = i; bestDx = dx; }
38- }
39- if (best >= 0) this.onMarkerClick?.(best);
40- });
31+ canvas.addEventListener('pointerdown', (e) => this.pointerDown(e));
32+ canvas.addEventListener('pointermove', (e) => this.pointerMove(e));
33+ canvas.addEventListener('pointerup', (e) => this.pointerUp(e));
4134 new ResizeObserver(() => this.draw()).observe(canvas);
4235 }
4336
@@ -45,7 +38,77 @@ export class WaveView {
4538 this.clip = clip;
4639 this.markers = markers;
4740 this.color = color;
48- this.selected = -1;
41+ this.selection.clear();
42+ this.draw();
43+ }
44+
45+ /** Update markers without dropping the selection (indices unchanged). */
46+ updateMarkers(markers: Marker[]): void {
47+ this.markers = markers;
48+ this.draw();
49+ }
50+
51+ private markerX(i: number, w: number): number {
52+ const dur = this.clip ? this.clip.samples.length / this.clip.sr : 1;
53+ return (this.markers[i].t / dur) * w;
54+ }
55+
56+ private markerAt(x: number): number {
57+ const w = this.canvas.getBoundingClientRect().width;
58+ let best = -1, bestDx = 8;
59+ for (let i = 0; i < this.markers.length; i++) {
60+ const dx = Math.abs(this.markerX(i, w) - x);
61+ if (dx < bestDx) { best = i; bestDx = dx; }
62+ }
63+ return best;
64+ }
65+
66+ private pointerDown(e: PointerEvent): void {
67+ if (!this.clip) return;
68+ const x = e.clientX - this.canvas.getBoundingClientRect().left;
69+ const additive = e.shiftKey || e.ctrlKey || e.metaKey;
70+ const i = this.markerAt(x);
71+ if (i >= 0) {
72+ if (additive) {
73+ if (this.selection.has(i)) this.selection.delete(i);
74+ else this.selection.add(i);
75+ } else {
76+ this.selection = new Set([i]);
77+ this.onAudition?.(i);
78+ }
79+ this.onSelectionChange?.();
80+ this.draw();
81+ } else {
82+ this.band = { x0: x, x1: x, base: additive ? new Set(this.selection) : new Set(), moved: false, additive };
83+ this.canvas.setPointerCapture(e.pointerId);
84+ }
85+ }
86+
87+ private pointerMove(e: PointerEvent): void {
88+ if (!this.band) return;
89+ const x = e.clientX - this.canvas.getBoundingClientRect().left;
90+ this.band.x1 = x;
91+ if (Math.abs(x - this.band.x0) > 3) this.band.moved = true;
92+ const lo = Math.min(this.band.x0, this.band.x1);
93+ const hi = Math.max(this.band.x0, this.band.x1);
94+ const w = this.canvas.getBoundingClientRect().width;
95+ this.selection = new Set(this.band.base);
96+ for (let i = 0; i < this.markers.length; i++) {
97+ const mx = this.markerX(i, w);
98+ if (mx >= lo && mx <= hi) this.selection.add(i);
99+ }
100+ this.onSelectionChange?.();
101+ this.draw();
102+ }
103+
104+ private pointerUp(e: PointerEvent): void {
105+ if (!this.band) return;
106+ if (!this.band.moved && !this.band.additive && this.selection.size) {
107+ this.selection.clear(); // a plain click on empty space clears
108+ this.onSelectionChange?.();
109+ }
110+ this.band = null;
111+ if (this.canvas.hasPointerCapture(e.pointerId)) this.canvas.releasePointerCapture(e.pointerId);
49112 this.draw();
50113 }
51114
@@ -64,8 +127,7 @@ export class WaveView {
64127 g.fillRect(0, 0, w, h);
65128 if (!this.clip) return;
66129
67- const { samples, sr } = this.clip;
68- const dur = samples.length / sr;
130+ const { samples } = this.clip;
69131 const mid = h / 2;
70132 g.strokeStyle = WAVE;
71133 g.lineWidth = 1;
@@ -85,9 +147,10 @@ export class WaveView {
85147
86148 for (let i = 0; i < this.markers.length; i++) {
87149 const m = this.markers[i];
88- const x = (m.t / dur) * w;
150+ const x = this.markerX(i, w);
151+ const sel = this.selection.has(i);
89152 g.strokeStyle = this.color;
90- g.lineWidth = i === this.selected ? 2.5 : 1.2;
153+ g.lineWidth = sel ? 2.5 : 1.2;
91154 g.globalAlpha = m.excluded ? 0.35 : 1;
92155 g.beginPath();
93156 g.moveTo(x, 2);
@@ -97,7 +160,22 @@ export class WaveView {
97160 g.arc(x, 8, 4.5, 0, 2 * Math.PI);
98161 if (m.excluded) g.stroke();
99162 else { g.fillStyle = this.color; g.fill(); }
163+ if (sel) {
164+ g.strokeStyle = '#fff';
165+ g.lineWidth = 1.5;
166+ g.beginPath();
167+ g.arc(x, 8, 6.5, 0, 2 * Math.PI);
168+ g.stroke();
169+ }
100170 g.globalAlpha = 1;
101171 }
172+
173+ if (this.band) {
174+ const lo = Math.min(this.band.x0, this.band.x1);
175+ g.fillStyle = 'rgba(120, 170, 255, 0.15)';
176+ g.strokeStyle = 'rgba(120, 170, 255, 0.5)';
177+ g.fillRect(lo, 0, Math.abs(this.band.x1 - this.band.x0), h);
178+ g.strokeRect(lo + 0.5, 0.5, Math.abs(this.band.x1 - this.band.x0) - 1, h - 1);
179+ }
102180 }
103181 }
tsconfig.jsonmodified+10−3View file
@@ -9,7 +9,14 @@
99 "noUncheckedIndexedAccess": false,
1010 "noEmit": true,
1111 "skipLibCheck": true,
12- "types": ["vite/client", "node"]
12+ "types": [
13+ "@webgpu/types",
14+ "vite/client",
15+ "node"
16+ ]
1317 },
14- "include": ["src", "scripts"]
15-}
18+ "include": [
19+ "src",
20+ "scripts"
21+ ]
22+}
vite.config.tsmodified+9−0View file
@@ -1,6 +1,15 @@
11 import { defineConfig } from 'vite';
2+import { realpathSync } from 'node:fs';
3+import { resolve } from 'node:path';
4+
5+// instrawave is a file: dependency on the sibling checkout (the vault
6+// layout); it is TypeScript source behind a symlink, so it must be served
7+// from its real path and kept out of prebundling.
8+const instrawave = realpathSync(resolve(import.meta.dirname, '../instrawave'));
29
310 export default defineConfig({
411 base: './',
12+ optimizeDeps: { exclude: ['instrawave'] },
13+ server: { fs: { allow: [import.meta.dirname, instrawave] } },
514 build: { target: 'es2022' },
615 });