Tuning audition, instrument geometry editor with 3D view, full browser persistence - proof-of-concept/instrawave-compose

proof-of-concept / instrawave-compose

Tuning audition, instrument geometry editor with 3D view, full browser persistence

A Hear button in each voice's tuning panel renders one hit at the current tuning from the instrument's bake. The bake bar gains an Instrument panel per bound instrument: instrawave's raymarched scene viewer next to the geometry and accuracy sliders, with edits persisting per instrument and marking the bake stale. Everything now survives a reload: the take's audio and edited events, geometry overrides, slider positions, and the bakes themselves, serialized into IndexedDB and reloaded on first use.

Jeremy Magland <jmagland@flatironinstitute.org> committed 1 month ago commit 9f20d5bfab29 parent bd6f329 Browse files

5 changed files+476−85

README.mdmodified+19−8View file

@@ -41,8 +41,11 @@ be wrong, and the fix is one gesture: drag the marker to the right lane.

4141 "Learn" folds the corrected hit back into that voice's training set, so

4242 the labelling improves with use; there is no model to refit.

4343

44-Takes are not persisted; the take is a performance, and re-recording one

45-is the cheap operation. During playback each hit flashes as the playhead

44+Everything persists in the browser: voices and their training, the take

45+(audio and edited events), per-instrument geometry, slider positions, and

46+the bakes themselves, which round-trip through IndexedDB so a reload finds

47+them stored and loads one the first time it is needed rather than paying

48+the precompute again. During playback each hit flashes as the playhead

4649 crosses it, ring size following the hit's strength, so the dynamics are

4750 visible in motion as well as in the bar heights.

4851

@@ -55,9 +58,16 @@ chime) with per-voice tuning overrides, edited as sliders under the voice.

5558 Playback touches the GPU only when at least one bound instrument is

5659 instrawave; a purely synth arrangement never asks for a device.

5760

58-The instrawave instruments are held at their default geometry and

59-accuracy, so each has exactly one bake, computed on demand from the bake

60-bar (with progress and a cancel) and kept for the session. All per-voice

61+Each instrument carries one set of geometry and accuracy values, edited

62+in the instrument panel that opens from the bake bar: instrawave's own

63+raymarched scene view alongside the geometry and accuracy sliders, so the

64+staircased sound hole or the microphone position is seen exactly as the

65+solver sees it, and any edit marks the bake stale in the bar. Geometry

66+belongs to the instrument rather than the voice because it is what the

67+shared bake depends on; each instrument therefore has exactly one bake,

68+computed on demand (with progress and a cancel). A Hear button in each

69+voice's tuning panel renders a single hit at the current tuning from that

70+bake, so tuning is auditioned without recording anything. All per-voice

6171 character travels as tuning, which is free: every voice bound to the same

6272 instrument shares its bake, and one render call carries all their events,

6373 each hit an event `{t, set: {...tuning, vel}}` with the voice's overrides

@@ -84,9 +94,10 @@ unfamiliar sound lands on whatever is closest. Overlapping hits closer

8494 than 50 ms merge into one. The browser's microphone processing is asked to

8595 stand down (automatic gain control would rewrite the very dynamics being

8696 tracked), but a system-level AGC outside the browser will still flatten

87-them. Bakes are not persisted (the drum's is a couple of hundred MB), so

88-the first instrawave playback of a session pays the precompute, about 8 s

89-for the drum on the development machine. A hit learned from a take stores

97+them. A stored bake is a couple of hundred MB in IndexedDB for the drum;

98+browsers may evict it under storage pressure, in which case the precompute

99+(about 8 s for the drum on the development machine) simply runs again. A

100+hit learned from a take stores

90101 only its features, so it cannot be re-auditioned later; clips can. A

91102 multi-event instrawave render costs one synthesis pass per event over the

92103 whole trace, so a very dense take renders slowly.

index.htmlmodified+28−0View file

@@ -41,6 +41,17 @@

4141 .tuning-panel label span:last-child { opacity: 0.7; font-variant-numeric: tabular-nums; }

4242 .tuning-panel input[type='range'] { width: 100%; }

4343 #bakebar progress { width: 8rem; }

44+ #iwedit { border: 1px solid rgba(128,128,128,0.35); border-radius: 8px; padding: 0.6rem 0.8rem; margin: 0.5rem 0; }

45+ .iwedit-grid { display: grid; grid-template-columns: minmax(0, 1fr) 24rem; gap: 0 1.5rem; align-items: start; }

46+ @media (max-width: 56rem) { .iwedit-grid { grid-template-columns: 1fr; } }

47+ .params { display: grid; grid-template-columns: repeat(auto-fill, minmax(14rem, 1fr)); gap: 0.3rem 1.5rem; }

48+ .params .param { display: grid; font-size: 0.82rem; }

49+ .params label { display: flex; justify-content: space-between; }

50+ .params label span:last-child { opacity: 0.7; font-variant-numeric: tabular-nums; }

51+ .params input[type='range'] { width: 100%; }

52+ .scene { position: relative; width: 24rem; max-width: 100%; aspect-ratio: 4 / 3; }

53+ .scene canvas { position: absolute; inset: 0; width: 100%; height: 100%; border-radius: 6px; }

54+ .scene .scene-hint { position: absolute; right: 0.5rem; bottom: 0.3rem; font-size: 0.7rem; color: rgba(255,255,255,0.45); }

4455 .voice-row .spacer { flex: 1; }

4556 .rec { color: #d33; }

4657 canvas.wave { width: 100%; height: 110px; border-radius: 6px; display: block; touch-action: none; }

@@ -96,6 +107,23 @@

96107 <button id="take-play" disabled>Play</button>

97108 </div>

98109 <div class="bar" id="bakebar" hidden></div>

110+

111+ <div id="iwedit" hidden>

112+ <div class="bar">

113+ <strong id="iwedit-title"></strong>

114+ <span class="hint">geometry and accuracy feed the bake; changing one makes it stale · shared by every voice bound to this instrument</span>

115+ <span style="flex:1"></span>

116+ <button id="iwedit-done">Done</button>

117+ </div>

118+ <div class="iwedit-grid">

119+ <div class="params" id="iwedit-params"></div>

120+ <div class="scene">

121+ <canvas id="iw-volume"></canvas>

122+ <canvas id="iw-overlay"></canvas>

123+ <span class="hint scene-hint">drag to orbit · wheel to zoom</span>

124+ </div>

125+ </div>

126+ </div>

99127 <canvas id="timeline" class="tl"></canvas>

100128 <div class="bar">

101129 <span class="hint">drag a hit to another lane to reassign · double-click a lane to add a hit · delete key removes</span>

src/main.tsmodified+258−18View file

@@ -15,10 +15,13 @@ import { classify, type TrainingSet } from './analysis/classify.ts';

1515 import { renderSynth } from './render/synth.ts';

1616 import { gainFromPeakDb, type NoteEvent, type SynthPreset } from './render/types.ts';

1717 import { mixInto, peak, resampleLinear } from './render/mix.ts';

18-import { IwEngine, PhaseCancelled, type StimulusEvent } from './render/instrawave.ts';

18+import { IwEngine, parseBake, PhaseCancelled, type StimulusEvent } from './render/instrawave.ts';

19+import { Viewer, type Camera } from 'instrawave/src/host/view.ts';

1920 import {

20- clearClipAudio, clearStore, deleteClipAudio, getClipAudio, loadVoices, putClipAudio,

21- saveVoices, trainPool, type TrainClip, type Voice,

21+ clearBulk, clearStore, deleteClipAudio, getBakeBytes, getBakeKey, getClipAudio, getTakeAudio,

22+ loadIwOverrides, loadTakeHits, loadUi, loadVoices, putBake, putClipAudio, putTakeAudio,

23+ saveIwOverrides, saveTakeHits, saveUi, saveVoices, trainPool,

24+ type TakeHit, type TrainClip, type Voice,

2225 } from './store.ts';

2326 import { WaveView, type Marker } from './ui/waveform.ts';

2427 import { Timeline } from './ui/timeline.ts';

@@ -27,12 +30,6 @@ const PALETTE = ['#e4573d', '#3d7de4', '#3dbb6e', '#d9a520', '#9061d9', '#d95fa8

2730 const PRESETS: SynthPreset[] = ['kick', 'snare', 'hat', 'tom'];

2831 const MIX_SR = 44100;

2932

30-interface TakeHit {

31- id: string; t: number; voiceId: string; gain: number;

32- peakDb: number; vec: number[] | null; // null for hand-added hits

33- s0: number; s1: number; // sample bounds in the take clip; -1 if hand-added

34-}

35-

3633 const $ = <T extends HTMLElement>(id: string) => document.getElementById(id) as T;

3734 const uid = () => Math.random().toString(36).slice(2, 10);

3835

@@ -51,7 +48,11 @@ let playing: { src: AudioBufferSourceNode; t0: number; dur: number } | null = nu

5148 let renderCache: { sig: string; buf: AudioBuffer } | null = null;

5249

5350 const iw = new IwEngine();

51+iw.setOverrides(loadIwOverrides());

52+const saveIw = () => saveIwOverrides(iw.dumpOverrides());

53+const ui = loadUi();

5454 let bakeJob: { name: string; cancel: boolean } | null = null;

55+let iwBusy = false;

5556 let bakeProg: HTMLProgressElement | null = null;

5657

5758 function save(): void { saveVoices(voices); }

@@ -166,6 +167,8 @@ async function stopRecording(): Promise<void> {

166167 renderVoiceList();

167168 } else {

168169 take.clip = clip;

170+ void putTakeAudio(clip.samples, clip.sr).catch(() =>

171+ status('warning: take audio could not be stored; it will not survive a reload'));

169172 processTake();

170173 }

171174 }

@@ -273,6 +276,8 @@ waveView.onSelectionChange = syncEditorButtons;

273276 $('capture-exclude').addEventListener('click', toggleExcluded);

274277

275278 $('capture-sens').addEventListener('input', () => {

279+ ui.captureSens = Number(($('capture-sens') as HTMLInputElement).value);

280+ saveUi(ui);

276281 const clip = editingClip();

277282 if (!edit || !clip) return;

278283 clip.sens = Number(($('capture-sens') as HTMLInputElement).value);

@@ -421,6 +426,7 @@ function renderVoiceList(): void {

421426 for (const c of voice.clips) void deleteClipAudio(c.id);

422427 voices = voices.filter((v) => v.id !== voice.id);

423428 take.hits = take.hits.filter((h) => h.voiceId !== voice.id);

429+ saveTakeHits(take.hits);

424430 save();

425431 if (edit?.voiceId === voice.id) closeEditor();

426432 renderVoiceList();

@@ -442,6 +448,18 @@ function renderVoiceList(): void {

442448 function tuningPanel(voice: Voice, instrument: string, set: Record<string, number>): HTMLElement {

443449 const panel = document.createElement('div');

444450 panel.className = 'tuning-panel';

451+ const bar = document.createElement('div');

452+ bar.className = 'bar';

453+ bar.style.gridColumn = '1 / -1';

454+ bar.style.margin = '0';

455+ const hear = document.createElement('button');

456+ hear.textContent = 'Hear';

457+ hear.addEventListener('click', () => void auditionIw(instrument, set));

458+ const hint = document.createElement('span');

459+ hint.className = 'hint';

460+ hint.textContent = 'one hit at these settings, from the current bake';

461+ bar.append(hear, hint);

462+ panel.appendChild(bar);

445463 for (const p of iw.tuning(instrument)) {

446464 const div = document.createElement('div');

447465 div.className = 'param';

@@ -489,8 +507,11 @@ $('add-voice').addEventListener('click', () => {

489507 $('reset-all').addEventListener('click', () => {

490508 if (!confirm('Forget all voices, training, and the current take?')) return;

491509 clearStore();

492- void clearClipAudio();

510+ void clearBulk();

511+ iw.setOverrides({});

512+ saveIw();

493513 voices = [];

514+ closeIwEditor();

494515 take = { clip: null, hits: [] };

495516 closeEditor();

496517 renderVoiceList();

@@ -508,22 +529,38 @@ function neededInstruments(): string[] {

508529 }

509530

510531 function updateBakeBar(): void {

532+ void refreshBakeBar();

533+}

534+

535+let bakeBarEpoch = 0;

536+async function refreshBakeBar(): Promise<void> {

537+ const epoch = ++bakeBarEpoch;

511538 const bar = $('bakebar');

512539 const needed = neededInstruments();

513540 bar.hidden = needed.length === 0;

541+ if (needed.length === 0) { bar.textContent = ''; return; }

542+ const states = await Promise.all(needed.map(async (name) => {

543+ if (await iw.bakeReady(name)) return 'ready' as const;

544+ if (bakeJob?.name === name) return 'baking' as const;

545+ const stored = await getBakeKey(name).catch(() => null);

546+ return stored === (await iw.currentKey(name)) ? ('stored' as const) : ('none' as const);

547+ }));

548+ if (epoch !== bakeBarEpoch) return; // a newer refresh superseded this one

514549 bar.textContent = '';

515550 bakeProg = null;

516- for (const name of needed) {

551+ needed.forEach((name, i) => {

517552 const label = document.createElement('span');

518553 label.className = 'hint';

519554 label.textContent = `${name} bake:`;

520555 bar.appendChild(label);

521- if (iw.hasBake(name)) {

556+ const st = states[i];

557+ if (st === 'ready' || st === 'stored') {

522558 const ok = document.createElement('span');

523- ok.textContent = 'ready';

559+ ok.textContent = st;

524560 ok.style.color = '#15803d';

561+ if (st === 'stored') ok.title = 'saved in the browser; loads on first use';

525562 bar.appendChild(ok);

526- } else if (bakeJob?.name === name) {

563+ } else if (st === 'baking') {

527564 const prog = document.createElement('progress');

528565 prog.max = 1;

529566 prog.value = 0;

@@ -539,7 +576,29 @@ function updateBakeBar(): void {

539576 btn.addEventListener('click', () => void startBake(name));

540577 bar.appendChild(btn);

541578 }

542- }

579+ const inst = document.createElement('button');

580+ inst.textContent = 'Instrument\u2026';

581+ inst.title = 'geometry, accuracy, and the 3D view';

582+ inst.addEventListener('click', () => void openIwEditor(name));

583+ bar.appendChild(inst);

584+ });

585+}

586+

587+/** Bring a bake into memory: already there, or stored in the browser from

588+ * an earlier session under the same geometry. */

589+async function ensureBakeLoaded(name: string): Promise<boolean> {

590+ if (await iw.bakeReady(name)) return true;

591+ const key = await iw.currentKey(name);

592+ if ((await getBakeKey(name).catch(() => null)) !== key) return false;

593+ status(`loading the stored ${name} bake\u2026`);

594+ const bytes = await getBakeBytes(name).catch(() => null);

595+ if (!bytes) { status(''); return false; }

596+ const bake = parseBake(bytes);

597+ if (bake.key !== key) return false;

598+ iw.adoptBake(name, bake);

599+ status('');

600+ updateBakeBar();

601+ return true;

543602 }

544603

545604 async function startBake(name: string): Promise<void> {

@@ -552,7 +611,14 @@ async function startBake(name: string): Promise<void> {

552611 (done, total) => { if (bakeProg) bakeProg.value = done / total; },

553612 () => bakeJob?.cancel ?? true,

554613 );

555- status(`${name} bake ready`);

614+ status(`${name} bake ready; storing it in the browser\u2026`);

615+ const ser = iw.serializedBake(name);

616+ if (ser) {

617+ await putBake(name, ser.key, ser.bytes).then(

618+ () => status(`${name} bake ready and stored (${(ser.bytes.length / 1e6).toFixed(0)} MB)`),

619+ () => status(`${name} bake ready, but storing it failed; it will be recomputed next session`),

620+ );

621+ }

556622 } catch (e) {

557623 status(e instanceof PhaseCancelled ? `${name} precompute cancelled`

558624 : `${name} precompute failed: ${e instanceof Error ? e.message : e}`);

@@ -592,6 +658,7 @@ function processTake(): void {

592658 }

593659 for (const h of hits) h.gain = gainFromPeakDb(h.peakDb, refDb);

594660 take.hits = hits;

661+ saveTakeHits(take.hits);

595662 refreshTimeline();

596663 syncTakeBar();

597664 if (voices.length === 0) status('no voices defined; add and train a voice, then hit re-detect');

@@ -635,7 +702,11 @@ $('take-record').addEventListener('click', () => {

635702 });

636703

637704 $('take-redetect').addEventListener('click', processTake);

638-$('take-sens').addEventListener('change', () => { if (take.clip) processTake(); });

705+$('take-sens').addEventListener('change', () => {

706+ ui.takeSens = Number(($('take-sens') as HTMLInputElement).value);

707+ saveUi(ui);

708+ if (take.clip) processTake();

709+});

639710

640711 timeline.onSelect = (id) => {

641712 syncHitButtons();

@@ -653,6 +724,7 @@ async function auditionSynth(hit: TakeHit, preset: SynthPreset): Promise<void> {

653724 }

654725

655726 timeline.onReassign = (id, voiceId) => {

727+ saveTakeHits(take.hits);

656728 const voice = voiceById(voiceId);

657729 status(voice ? `hit moved to "${voice.name}"; Learn adds it to that voice's training` : '');

658730 syncHitButtons();

@@ -661,6 +733,7 @@ timeline.onReassign = (id, voiceId) => {

661733 timeline.onAdd = (t, voiceId) => {

662734 take.hits.push({ id: uid(), t, voiceId, gain: 0.8, peakDb: 0, vec: null, s0: -1, s1: -1 });

663735 take.hits.sort((a, b) => a.t - b.t);

736+ saveTakeHits(take.hits);

664737 refreshTimeline();

665738 };

666739

@@ -679,6 +752,7 @@ document.addEventListener('keydown', (e) => {

679752 function deleteSelected(): void {

680753 if (!timeline.selected) return;

681754 take.hits = take.hits.filter((h) => h.id !== timeline.selected);

755+ saveTakeHits(take.hits);

682756 timeline.selected = null;

683757 refreshTimeline();

684758 }

@@ -701,6 +775,7 @@ async function renderMix(): Promise<AudioBuffer> {

701775 const sig = JSON.stringify({

702776 hits: take.hits.map((h) => [h.t, h.voiceId, h.gain]),

703777 bindings: voices.map((v) => [v.id, v.binding]),

778+ iw: iw.dumpOverrides(),

704779 });

705780 if (renderCache?.sig === sig) return renderCache.buf;

706781

@@ -723,7 +798,10 @@ async function renderMix(): Promise<AudioBuffer> {

723798 list.push({ t: h.t, set });

724799 }

725800 }

726- const missing = [...iwEvents.keys()].filter((n) => !iw.hasBake(n));

801+ const missing: string[] = [];

802+ for (const n of iwEvents.keys()) {

803+ if (!(await ensureBakeLoaded(n))) missing.push(n);

804+ }

727805 if (missing.length) {

728806 throw new Error(`precompute the ${missing.join(' and ')} bake${missing.length > 1 ? 's' : ''} first`);

729807 }

@@ -776,8 +854,170 @@ $('take-play').addEventListener('click', async () => {

776854 }

777855 });

778856

857+// ---- tuning audition ----------------------------------------------------------

858+

859+async function auditionIw(instrument: string, set: Record<string, number>): Promise<void> {

860+ if (iwBusy || bakeJob) { status('the GPU is busy; try again in a moment'); return; }

861+ iwBusy = true;

862+ try {

863+ if (!(await ensureBakeLoaded(instrument))) {

864+ status(`no current ${instrument} bake; precompute it in the bake bar below`);

865+ return;

866+ }

867+ const allowed = new Set(iw.tuning(instrument).map((p) => p.key));

868+ const ev: Record<string, number> = {};

869+ for (const [k, v] of Object.entries(set)) if (allowed.has(k)) ev[k] = v;

870+ const t60 = set.t60 ?? (allowed.has('t60') ? iw.defaultValue(instrument, 't60') : 2);

871+ const seconds = Math.min(5, Math.max(1.2, t60 * 0.6 + 0.5));

872+ const { trace, dt } = await iw.render(instrument, seconds, [{ t: 0, set: ev }], (done, total) =>

873+ status(`rendering ${instrument}: ${Math.round((100 * done) / total)}%`));

874+ const res = resampleLinear(trace, 1 / dt, MIX_SR);

875+ const pk = peak(res);

876+ if (pk > 0) for (let i = 0; i < res.length; i++) res[i] *= 0.8 / pk;

877+ const buf = audio().createBuffer(1, res.length, MIX_SR);

878+ buf.getChannelData(0).set(res);

879+ status('');

880+ void playBuffer(buf);

881+ } catch (e) {

882+ status(e instanceof Error ? e.message : String(e));

883+ } finally {

884+ iwBusy = false;

885+ }

886+}

887+

888+// ---- instrument editor (geometry, accuracy, the 3D view) ----------------------

889+// Geometry belongs to the instrument, not the voice: it is what the shared

890+// bake depends on, so the editor opens from the bake bar and an edit marks

891+// the bake stale there. The scene is instrawave's own viewer, raymarching

892+// the solver's actual fields, so what it shows is what the bake will see.

893+

894+let iwEdit: { name: string; viewer: Viewer | null } | null = null;

895+const iwCam: Camera = { yaw: -2.2, pitch: 0.45, dist: 1.6 };

896+const iwVolume = $('iw-volume') as HTMLCanvasElement;

897+const iwOverlay = $('iw-overlay') as HTMLCanvasElement;

898+

899+function sizeSceneCanvas(c: HTMLCanvasElement): void {

900+ const r = c.getBoundingClientRect();

901+ const dpr = Math.min(devicePixelRatio || 1, 2);

902+ const w = Math.max(1, Math.round(r.width * dpr));

903+ const h = Math.max(1, Math.round(r.height * dpr));

904+ if (c.width !== w || c.height !== h) { c.width = w; c.height = h; }

905+}

906+

907+let iwViewTimer: number | null = null;

908+function scheduleIwView(): void {

909+ if (iwViewTimer !== null) clearTimeout(iwViewTimer);

910+ iwViewTimer = window.setTimeout(() => {

911+ iwViewTimer = null;

912+ const e = iwEdit;

913+ if (!e?.viewer) return;

914+ void (async () => {

915+ try {

916+ await e.viewer!.update(iw.scope(e.name));

917+ e.viewer!.draw(iwCam);

918+ } catch (err) {

919+ status(`scene view: ${err instanceof Error ? err.message : err}`);

920+ }

921+ })();

922+ }, 180);

923+}

924+

925+async function openIwEditor(name: string): Promise<void> {

926+ iwEdit = { name, viewer: null };

927+ $('iwedit').hidden = false;

928+ $('iwedit-title').textContent = name;

929+ buildIwParams(name);

930+ try {

931+ const device = await iw.ensureDevice();

932+ if (iwEdit?.name !== name) return;

933+ sizeSceneCanvas(iwVolume);

934+ sizeSceneCanvas(iwOverlay);

935+ iwEdit.viewer = new Viewer(device, iw.file(name), iwVolume, iwOverlay);

936+ scheduleIwView();

937+ } catch (e) {

938+ status(`scene view unavailable: ${e instanceof Error ? e.message : e} (the sliders still work)`);

939+ }

940+}

941+

942+function buildIwParams(name: string): void {

943+ const box = $('iwedit-params');

944+ box.textContent = '';

945+ const values = iw.values(name);

946+ for (const p of iw.geometryParams(name)) {

947+ const div = document.createElement('div');

948+ div.className = 'param';

949+ const label = document.createElement('label');

950+ const nameSpan = document.createElement('span');

951+ nameSpan.textContent = `${p.label} (${p.unit})`;

952+ if (p.hint) nameSpan.title = p.hint;

953+ const valSpan = document.createElement('span');

954+ const input = document.createElement('input');

955+ input.type = 'range';

956+ input.min = String(p.min);

957+ input.max = String(p.max);

958+ input.step = String(p.step ?? (p.max - p.min) / 200);

959+ input.value = String(values[p.key]);

960+ const show = () => { valSpan.textContent = Number(input.value).toPrecision(4); };

961+ show();

962+ input.addEventListener('input', () => {

963+ iw.setOverride(name, p.key, Number(input.value));

964+ show();

965+ saveIw();

966+ scheduleIwView();

967+ updateBakeBar(); // the bake just went stale (or came back)

968+ });

969+ label.append(nameSpan, valSpan);

970+ div.append(label, input);

971+ box.appendChild(div);

972+ }

973+}

974+

975+function closeIwEditor(): void {

976+ iwEdit = null;

977+ $('iwedit').hidden = true;

978+}

979+$('iwedit-done').addEventListener('click', closeIwEditor);

980+

981+{

982+ let dragging = false, px = 0, py = 0;

983+ iwOverlay.addEventListener('pointerdown', (e) => {

984+ dragging = true; px = e.clientX; py = e.clientY;

985+ iwOverlay.setPointerCapture(e.pointerId);

986+ });

987+ iwOverlay.addEventListener('pointermove', (e) => {

988+ if (!dragging || !iwEdit?.viewer) return;

989+ iwCam.yaw -= (e.clientX - px) * 0.008;

990+ iwCam.pitch = Math.min(1.45, Math.max(-1.45, iwCam.pitch + (e.clientY - py) * 0.008));

991+ px = e.clientX; py = e.clientY;

992+ iwEdit.viewer.draw(iwCam);

993+ });

994+ iwOverlay.addEventListener('pointerup', () => { dragging = false; });

995+ iwOverlay.addEventListener('wheel', (e) => {

996+ if (!iwEdit?.viewer) return;

997+ e.preventDefault();

998+ iwCam.dist = Math.min(6, Math.max(0.3, iwCam.dist * Math.exp(e.deltaY * 0.001)));

999+ iwEdit.viewer.draw(iwCam);

1000+ }, { passive: false });

1001+ window.addEventListener('resize', () => {

1002+ if (!iwEdit?.viewer) return;

1003+ sizeSceneCanvas(iwVolume);

1004+ sizeSceneCanvas(iwOverlay);

1005+ scheduleIwView();

1006+ });

1007+}

1008+

7791009 // ---- boot -----------------------------------------------------------------

7801010

1011+($('capture-sens') as HTMLInputElement).value = String(ui.captureSens);

1012+($('take-sens') as HTMLInputElement).value = String(ui.takeSens);

1013+take.hits = loadTakeHits() ?? [];

1014+void getTakeAudio().then((a) => {

1015+ if (a) {

1016+ take.clip = { samples: a.samples, sr: a.sr };

1017+ refreshTimeline();

1018+ syncTakeBar();

1019+ }

1020+}).catch(() => { /* the hits still play and edit; re-detect needs the audio */ });

7811021 renderVoiceList();

7821022 refreshTimeline();

7831023 syncTakeBar();

src/render/instrawave.tsmodified+71−23View file

@@ -1,16 +1,19 @@

11 /**

2- * The instrawave side of the renderer seam. The three bundled instruments

3- * are held at their default geometry and accuracy, so each has exactly one

4- * bake, computed on demand and kept for the session (the drum's is a

5- * couple of hundred MB; persisting it is future work). All per-voice

6- * character travels as tuning, which is free: every voice bound to the

7- * same instrument shares its bake, and one render call carries all their

8- * events, each with the voice's tuning overrides and its strength scaling

9- * the instrument's excitation amplitude (`vel`, or `amp` where there is no

10- * strike).

2+ * The instrawave side of the renderer seam. Each of the three bundled

3+ * instruments carries one set of geometry and accuracy values (defaults

4+ * plus the user's overrides, edited in the instrument panel), so each has

5+ * exactly one live bake; the bake key tells a stale one from a current

6+ * one, and serialized bakes round-trip through IndexedDB so a reload does

7+ * not repeat the precompute. All per-voice character travels as tuning,

8+ * which is free: every voice bound to the same instrument shares its bake,

9+ * and one render call carries all their events, each with the voice's

10+ * tuning overrides and its strength scaling the instrument's excitation

11+ * amplitude (`vel`, or `amp` where there is no strike).

1112 */

1213 import { requestDevice } from 'instrawave/src/host/device.ts';

1314 import { InstrumentSession } from 'instrawave/src/host/session.ts';

15+import { bakeKey, serializeBake, type Bake } from 'instrawave/src/host/bake.ts';

16+import { evalDerived } from 'instrawave/src/expr.ts';

1417 import {

1518 parseInstrument,

1619 paramValues,

@@ -18,13 +21,13 @@ import {

1821 type ParamSpec,

1922 type StimulusEvent,

2023 } from 'instrawave/src/format.ts';

21-import type { Bake } from 'instrawave/src/host/bake.ts';

2224 import drumJson from 'instrawave/instruments/drum.instrument.json';

2325 import dulcimerJson from 'instrawave/instruments/dulcimer.instrument.json';

2426 import chimeJson from 'instrawave/instruments/chime.instrument.json';

2527

2628 export { PhaseCancelled } from 'instrawave/src/host/session.ts';

27-export type { StimulusEvent };

29+export { parseBake } from 'instrawave/src/host/bake.ts';

30+export type { StimulusEvent, Bake };

2831

2932 const files: InstrumentFile[] = [drumJson, dulcimerJson, chimeJson].map((j) =>

3033 parseInstrument(JSON.stringify(j)),

@@ -36,12 +39,13 @@ export class IwEngine {

3639 private device: GPUDevice | null = null;

3740 private bakes = new Map<string, Bake>();

3841 private insts = new Map(files.map((f) => [f.name, f]));

42+ private overrides: Record<string, Record<string, number>> = {};

3943

4044 names(): string[] {

4145 return [...this.insts.keys()];

4246 }

4347

44- private inst(name: string): InstrumentFile {

48+ file(name: string): InstrumentFile {

4549 const f = this.insts.get(name);

4650 if (!f) throw new Error(`unknown instrument '${name}'`);

4751 return f;

@@ -50,32 +54,76 @@ export class IwEngine {

5054 /** The tuning parameters a binding may override: tuning-class params

5155 * that are also event parameters, so they can vary per hit. */

5256 tuning(name: string): ParamSpec[] {

53- const f = this.inst(name);

57+ const f = this.file(name);

5458 return f.params.filter((p) => p.class === 'tuning' && f.render.eventParams.includes(p.key));

5559 }

5660

61+ /** Geometry and accuracy: the parameters the bake depends on. */

62+ geometryParams(name: string): ParamSpec[] {

63+ return this.file(name).params.filter((p) => p.class !== 'tuning');

64+ }

65+

5766 /** The parameter a hit's strength scales: the strike speed where there

5867 * is one, the pluck amplitude otherwise. Linear equations, so it is a

5968 * pure scale. */

6069 strengthParam(name: string): string {

61- const ep = this.inst(name).render.eventParams;

70+ const ep = this.file(name).render.eventParams;

6271 return ep.includes('vel') ? 'vel' : 'amp';

6372 }

6473

6574 defaultValue(name: string, key: string): number {

66- const p = this.inst(name).params.find((q) => q.key === key);

75+ const p = this.file(name).params.find((q) => q.key === key);

6776 if (!p) throw new Error(`instrument '${name}' has no parameter '${key}'`);

6877 return p.value;

6978 }

7079

71- hasBake(name: string): boolean {

72- return this.bakes.has(name);

80+ setOverrides(all: Record<string, Record<string, number>>): void {

81+ this.overrides = all;

82+ }

83+ setOverride(name: string, key: string, value: number): void {

84+ (this.overrides[name] ??= {})[key] = value;

85+ }

86+ dumpOverrides(): Record<string, Record<string, number>> {

87+ return this.overrides;

88+ }

89+

90+ values(name: string): Record<string, number> {

91+ return paramValues(this.file(name), this.overrides[name] ?? {});

92+ }

93+

94+ /** Params plus derived, fully evaluated (what the viewer draws from). */

95+ scope(name: string): Record<string, number> {

96+ const f = this.file(name);

97+ return evalDerived(f.derived, this.values(name));

98+ }

99+

100+ currentKey(name: string): Promise<string> {

101+ return bakeKey(this.file(name), this.values(name));

102+ }

103+

104+ async ensureDevice(): Promise<GPUDevice> {

105+ return (this.device ??= await requestDevice());

106+ }

107+

108+ /** True when an in-memory bake matches the current geometry. */

109+ async bakeReady(name: string): Promise<boolean> {

110+ const b = this.bakes.get(name);

111+ return !!b && b.key === (await this.currentKey(name));

112+ }

113+

114+ adoptBake(name: string, bake: Bake): void {

115+ this.bakes.set(name, bake);

116+ }

117+

118+ serializedBake(name: string): { key: string; bytes: Uint8Array } | null {

119+ const b = this.bakes.get(name);

120+ return b ? { key: b.key, bytes: serializeBake(b) } : null;

73121 }

74122

75123 async precompute(name: string, onProgress?: Progress, cancelled?: () => boolean): Promise<void> {

76- const inst = this.inst(name);

77- this.device ??= await requestDevice();

78- const session = new InstrumentSession(this.device, inst, paramValues(inst));

124+ const inst = this.file(name);

125+ const device = await this.ensureDevice();

126+ const session = new InstrumentSession(device, inst, this.values(name));

79127 this.bakes.set(name, await session.precompute(onProgress, cancelled));

80128 }

81129

@@ -85,11 +133,11 @@ export class IwEngine {

85133 events: StimulusEvent[],

86134 onProgress?: Progress,

87135 ): Promise<{ trace: Float32Array; dt: number }> {

88- const inst = this.inst(name);

136+ const inst = this.file(name);

89137 const bake = this.bakes.get(name);

90138 if (!bake) throw new Error(`no bake for '${name}'; precompute first`);

91- if (!this.device) throw new Error('no GPU device');

92- const session = new InstrumentSession(this.device, inst, paramValues(inst));

139+ const device = await this.ensureDevice();

140+ const session = new InstrumentSession(device, inst, this.values(name));

93141 const { trace, dt } = await session.render(bake, seconds, events, [], onProgress);

94142 return { trace, dt };

95143 }

src/store.tsmodified+100−36View file

@@ -1,10 +1,12 @@

11 /**

2- * Persistence. Voice metadata, training-clip descriptions, and cached

3- * feature vectors live in localStorage; the clips' raw audio lives in

4- * IndexedDB keyed by clip id, so a training can be reopened, re-detected,

5- * and re-labelled later. The two stores can drift only in one harmless

6- * direction: a clip whose audio is missing simply cannot be re-edited,

7- * while its cached features keep classifying.

2+ * Persistence. Small structured state lives in localStorage: voices with

3+ * their training-clip descriptions and cached features, per-instrument

4+ * geometry overrides, the take's events, and UI knobs. Bulk audio and

5+ * binary data live in IndexedDB: training-clip audio, the take's audio,

6+ * and the bakes (the drum's is a couple of hundred MB, which is exactly

7+ * why it is worth keeping). The stores can drift only in harmless

8+ * directions: a clip whose audio is missing cannot be re-edited but keeps

9+ * classifying, and a bake whose key no longer matches is ignored.

810 */

911 import type { Binding } from './render/types.ts';

1012

@@ -25,6 +27,11 @@ export interface Voice {

2527 clips: TrainClip[];

2628 loose: TrainHit[]; // hits learned from takes; they belong to no clip

2729 }

30+export interface TakeHit {

31+ id: string; t: number; voiceId: string; gain: number;

32+ peakDb: number; vec: number[] | null; // null for hand-added hits

33+ s0: number; s1: number; // sample bounds in the take clip; -1 if hand-added

34+}

2835

2936 /** The pooled training set a voice contributes to classification. */

3037 export function trainPool(v: Voice): TrainHit[] {

@@ -33,25 +40,34 @@ export function trainPool(v: Voice): TrainHit[] {

3340

3441 const KEY = 'instrawave-compose/v2';

3542 const KEY_V1 = 'instrawave-compose/v1';

43+const KEY_IW = 'instrawave-compose/iw';

44+const KEY_TAKE = 'instrawave-compose/take';

45+const KEY_UI = 'instrawave-compose/ui';

3646

37-export function loadVoices(): Voice[] {

47+function read<T>(key: string): T | null {

3848 try {

39- const raw = localStorage.getItem(KEY);

40- if (raw) return (JSON.parse(raw) as { voices: Voice[] }).voices;

41- const old = localStorage.getItem(KEY_V1);

42- if (old) {

43- // v1 stored a flat training list and a synth preset per voice.

44- const v1 = JSON.parse(old) as { voices: { id: string; name: string; color: string; preset: string; train: TrainHit[] }[] };

45- const voices: Voice[] = v1.voices.map((v) => ({

46- id: v.id, name: v.name, color: v.color,

47- binding: { kind: 'synth', preset: v.preset as ('kick' | 'snare' | 'hat' | 'tom') },

48- clips: [], loose: v.train,

49- }));

50- localStorage.setItem(KEY, JSON.stringify({ voices }));

51- localStorage.removeItem(KEY_V1);

52- return voices;

53- }

54- } catch { /* a corrupt store starts fresh */ }

49+ const raw = localStorage.getItem(key);

50+ return raw ? (JSON.parse(raw) as T) : null;

51+ } catch {

52+ return null;

53+ }

54+}

55+

56+export function loadVoices(): Voice[] {

57+ const doc = read<{ voices: Voice[] }>(KEY);

58+ if (doc) return doc.voices;

59+ // v1 stored a flat training list and a synth preset per voice.

60+ const v1 = read<{ voices: { id: string; name: string; color: string; preset: string; train: TrainHit[] }[] }>(KEY_V1);

61+ if (v1) {

62+ const voices: Voice[] = v1.voices.map((v) => ({

63+ id: v.id, name: v.name, color: v.color,

64+ binding: { kind: 'synth', preset: v.preset as ('kick' | 'snare' | 'hat' | 'tom') },

65+ clips: [], loose: v.train,

66+ }));

67+ localStorage.setItem(KEY, JSON.stringify({ voices }));

68+ localStorage.removeItem(KEY_V1);

69+ return voices;

70+ }

5571 return [];

5672 }

5773

@@ -59,43 +75,91 @@ export function saveVoices(voices: Voice[]): void {

5975 localStorage.setItem(KEY, JSON.stringify({ voices }));

6076 }

6177

78+/** Per-instrument geometry and accuracy overrides. */

79+export function loadIwOverrides(): Record<string, Record<string, number>> {

80+ return read(KEY_IW) ?? {};

81+}

82+export function saveIwOverrides(o: Record<string, Record<string, number>>): void {

83+ localStorage.setItem(KEY_IW, JSON.stringify(o));

84+}

85+

86+export function loadTakeHits(): TakeHit[] | null {

87+ return read(KEY_TAKE);

88+}

89+export function saveTakeHits(hits: TakeHit[]): void {

90+ localStorage.setItem(KEY_TAKE, JSON.stringify(hits));

91+}

92+

93+export interface UiState { takeSens: number; captureSens: number; }

94+export function loadUi(): UiState {

95+ return read(KEY_UI) ?? { takeSens: 0.6, captureSens: 0.6 };

96+}

97+export function saveUi(ui: UiState): void {

98+ localStorage.setItem(KEY_UI, JSON.stringify(ui));

99+}

100+

62101 export function clearStore(): void {

63- localStorage.removeItem(KEY);

102+ for (const k of [KEY, KEY_IW, KEY_TAKE, KEY_UI]) localStorage.removeItem(k);

64103 }

65104

66-// ---- clip audio in IndexedDB ------------------------------------------------

105+// ---- bulk data in IndexedDB ---------------------------------------------------

106+

107+const STORES = ['clips', 'takes', 'bakes', 'bakemeta'] as const;

67108

68109 let dbPromise: Promise<IDBDatabase> | null = null;

69110 function db(): Promise<IDBDatabase> {

70111 return (dbPromise ??= new Promise((resolve, reject) => {

71- const req = indexedDB.open('instrawave-compose', 1);

72- req.onupgradeneeded = () => req.result.createObjectStore('clips');

112+ const req = indexedDB.open('instrawave-compose', 2);

113+ req.onupgradeneeded = () => {

114+ for (const s of STORES) {

115+ if (!req.result.objectStoreNames.contains(s)) req.result.createObjectStore(s);

116+ }

117+ };

73118 req.onsuccess = () => resolve(req.result);

74119 req.onerror = () => reject(req.error);

75120 }));

76121 }

77122

78-function op<T>(mode: IDBTransactionMode, fn: (store: IDBObjectStore) => IDBRequest<T>): Promise<T> {

123+function op<T>(store: string, mode: IDBTransactionMode, fn: (s: IDBObjectStore) => IDBRequest<T>): Promise<T> {

79124 return db().then((d) => new Promise<T>((resolve, reject) => {

80- const req = fn(d.transaction('clips', mode).objectStore('clips'));

125+ const req = fn(d.transaction(store, mode).objectStore(store));

81126 req.onsuccess = () => resolve(req.result);

82127 req.onerror = () => reject(req.error);

83128 }));

84129 }

85130

86131 export function putClipAudio(id: string, samples: Float32Array, sr: number): Promise<unknown> {

87- return op('readwrite', (s) => s.put({ samples, sr }, id));

132+ return op('clips', 'readwrite', (s) => s.put({ samples, sr }, id));

88133 }

89-

90134 export function getClipAudio(id: string): Promise<{ samples: Float32Array; sr: number } | null> {

91- return op<{ samples: Float32Array; sr: number } | undefined>('readonly', (s) => s.get(id))

135+ return op<{ samples: Float32Array; sr: number } | undefined>('clips', 'readonly', (s) => s.get(id))

92136 .then((r) => r ?? null);

93137 }

94-

95138 export function deleteClipAudio(id: string): Promise<unknown> {

96- return op('readwrite', (s) => s.delete(id));

139+ return op('clips', 'readwrite', (s) => s.delete(id));

140+}

141+

142+export function putTakeAudio(samples: Float32Array, sr: number): Promise<unknown> {

143+ return op('takes', 'readwrite', (s) => s.put({ samples, sr }, 'take'));

144+}

145+export function getTakeAudio(): Promise<{ samples: Float32Array; sr: number } | null> {

146+ return op<{ samples: Float32Array; sr: number } | undefined>('takes', 'readonly', (s) => s.get('take'))

147+ .then((r) => r ?? null);

148+}

149+

150+/** Bake bytes and their key are separate stores so staleness can be

151+ * checked without deserializing hundreds of MB. */

152+export async function putBake(name: string, key: string, bytes: Uint8Array): Promise<void> {

153+ await op('bakes', 'readwrite', (s) => s.put(bytes, name));

154+ await op('bakemeta', 'readwrite', (s) => s.put(key, name));

155+}

156+export function getBakeKey(name: string): Promise<string | null> {

157+ return op<string | undefined>('bakemeta', 'readonly', (s) => s.get(name)).then((r) => r ?? null);

158+}

159+export function getBakeBytes(name: string): Promise<Uint8Array | null> {

160+ return op<Uint8Array | undefined>('bakes', 'readonly', (s) => s.get(name)).then((r) => r ?? null);

97161 }

98162

99-export function clearClipAudio(): Promise<unknown> {

100- return op('readwrite', (s) => s.clear());

163+export async function clearBulk(): Promise<void> {

164+ for (const s of STORES) await op(s, 'readwrite', (st) => st.clear());

101165 }