Widget: spectrogram of the sound (light entry loaded when near the viewport), fmax attribute and choice; axes and spectrogram read colors at their canvas - proof-of-concept/toy-instruments

proof-of-concept / toy-instruments

Widget: spectrogram of the sound (light entry loaded when near the viewport), fmax attribute and choice; axes and spectrogram read colors at their canvas

Jeremy Magland <jmagland@flatironinstitute.org> committed 11 days ago commit f89ae610457c parent d3b29d4 Browse files

5 changed files+66−15

README.mdmodified+2−2View file

@@ -77,8 +77,8 @@ Runs are cached on [artifact-cache](https://artifact-cache.tempory.net) (app `to

7777 <toy-instrument-sim name="crash" exaggeration="5" settings="https://magland.github.io/toy-instruments/#/bead-cymbal?…&preset=0&gpu=1"></toy-instrument-sim>

7878 ```

7979

80-- `settings` is a link from "Copy link to these settings" (or its `#/<instrument>?…` part). `exaggeration` magnifies the drawn motion (default 5; the reader can change it in the motion bar, from ×1 to ×100); it is not part of the cache key. `name` names the downloaded `.wav`.

81-- The reader sees the model at rest (the 3D view without its tabs, view buttons or exaggeration input; drag to rotate) and two buttons. "Play sound" downloads the light entry and plays it in an audio player; "Play motion (N MB)" downloads the frames and plays the replay in slow motion, with pause, a time slider, an exaggeration choice and a speed choice. Each row expands in place, so the other button does not move. On load the widget only sends HEAD requests (for the link's backend first, then the other).

80+- `settings` is a link from "Copy link to these settings" (or its `#/<instrument>?…` part). `exaggeration` magnifies the drawn motion (default 5; the reader can change it in the motion bar, from ×1 to ×100); it is not part of the cache key. `fmax` sets the top of the spectrogram's frequency axis (default 5000 Hz; the reader can change it). `name` names the downloaded `.wav`.

81+- The reader sees the model at rest (the 3D view without its tabs, view buttons or exaggeration input; drag to rotate), the spectrogram of the sound, and two buttons. The light entry is downloaded when the widget comes within 400 px of the viewport, for the spectrogram. "Play sound" plays it in an audio player; "Play motion (N MB)" downloads the frames and plays the replay in slow motion, with pause, a time slider, an exaggeration choice and a speed choice. Each row expands in place, so the other button does not move. On load the widget only sends HEAD requests (for the link's backend first, then the other); the frames are downloaded only for the motion.

8282 - If the cache has no result, or cannot be reached, the widget offers to run the simulation in the browser, with a rough time estimate, and then shows both players. With the cache passcode saved in that browser (under "Shared cache" in the widget), the run is stored under the same key.

8383 - "Open the full app" shows the whole instrument page (every parameter, the plots, Run and Load cached) over the host page, starting from the widget's settings; Close or Escape returns to the post, and reopening keeps its state. It is the bundle's own copy of the page (not the live app in a frame), so it computes the same cache keys as the widget. It has no back link or "Copy link", and leaves the address alone.

8484 - The key hashes what the bundle's own model code builds, so a page that keeps one build of the bundle asks for, and can always refill, the same entries, whatever later versions of the app do. jeremy.magland.org uploads each build under a content hash and never replaces it.

src/embed.cssmodified+3−0View file

@@ -51,6 +51,9 @@

5151 --diverging: #6da7ec, #1c5cab, #383835, #a83434, #e66767;

5252 }

5353 .scene-wrap { height: 340px; }

54+.spec-block { margin-top: 12px; }

55+.spec-head { display: flex; justify-content: space-between; align-items: baseline; gap: 8px; font-size: 13px; color: var(--text-secondary); }

56+.spec-wrap { height: 220px; margin-top: 4px; }

5457 @media (max-width: 600px) { .scene-wrap { height: 260px; } }

5558 .status { margin: 10px 0 0; color: var(--text-secondary); font-size: 13px; }

5659 .actions { display: flex; flex-wrap: wrap; gap: 8px; align-items: center; margin-top: 10px; }

src/embed.tsmodified+52−4View file

@@ -4,6 +4,7 @@ import SolverWorker from './sim/worker.ts?worker&inline';

44 import { CONFIGS, type AnyConfig } from './instruments/configs';

55 import { BeadScene } from './ui/scene3d';

66 import { RATES, Timeline } from './ui/timeline';

7+import { Spectrogram } from './ui/spectrogram';

78 import { encodeWav } from './ui/wav';

89 import { fmtNum, setStyleRoot } from './ui/plot';

910 import { mountInstrumentPage } from './ui/instrumentPage';

@@ -20,11 +21,13 @@ import type { SimResult } from './sim/types';

2021 // - exaggeration: how much the motion is magnified in the replay (default 5); the reader

2122 // can change it in the motion bar. It only affects the drawing, so it is not part of the

2223 // cache key.

24+// - fmax: top of the spectrogram's frequency axis in Hz (default 5000); the reader can change it.

2325 // - name: file name of the downloaded .wav (default toy-instrument).

2426 //

25-// The reader sees the 3D model with "Play sound" and "Play motion" buttons; each downloads

26-// what it needs from the shared cache on first press (the small light entry for the sound,

27-// the frames for the motion, played in slow motion). When the cache has no result, the

27+// The reader sees the 3D model, the spectrogram of the sound, and "Play sound" and "Play

28+// motion" buttons. The small light entry (audio and scalars) is downloaded when the widget

29+// comes near the viewport, for the spectrogram; the frames (most of the size) only when

30+// "Play motion" is first pressed, and played in slow motion. When the cache has no result, the

2831 // widget offers to run the simulation in the browser; with the cache passcode set, the run

2932 // is stored under the same key, so the next reader gets it from the cache. "Open the full

3033 // app" shows the whole instrument page (all parameters, plots, runs) over the host page,

@@ -54,6 +57,7 @@ class ToyInstrumentSim extends HTMLElement {

5457 private found: Found | null = null;

5558 private light: Promise<LightResult> | null = null;

5659 private exaggeration = 5;

60+ private spec!: Spectrogram;

5761 private cfg!: AnyConfig;

5862 private settings!: Settings<SimParams>;

5963 private overlayMounted = false;

@@ -65,6 +69,10 @@ class ToyInstrumentSim extends HTMLElement {

6569 <style>${baseCss}${embedCss}</style>

6670 <div class="tis">

6771 <div data-r="scene"></div>

72+ <div class="spec-block" data-r="spec-block" hidden>

73+ <div class="spec-head"><span>Spectrogram</span><label class="inline">up to <select data-r="fmax"></select> Hz</label></div>

74+ <div class="canvas-wrap spec-wrap"><canvas data-r="spec"></canvas></div>

75+ </div>

6876 <div class="players">

6977 <div class="player">

7078 <button class="btn play" data-r="sound-btn" hidden>▶ Play sound</button>

@@ -104,6 +112,7 @@ class ToyInstrumentSim extends HTMLElement {

104112 </div>`;

105113 this.mountPasscode();

106114 this.mountMotionBar();

115+ this.mountSpectrogram();

107116 this.start().catch((e) => this.fail(`Something went wrong: ${(e as Error).message}`));

108117 }

109118

@@ -190,10 +199,23 @@ class ToyInstrumentSim extends HTMLElement {

190199 runBtn.addEventListener('click', () => this.run(id, gpu));

191200 }

192201

193- // A stored result: buttons that download the sound and the motion when first pressed.

202+ // A stored result: the spectrogram once the widget is near the viewport, and buttons that

203+ // download the sound and the motion when first pressed.

194204 private offerPlayback() {

195205 const f = this.found!;

196206 this.status(f.stored ? `Computed on ${f.stored.toISOString().slice(0, 10)}.` : '');

207+ const near = new IntersectionObserver(

208+ (entries) => {

209+ if (!entries.some((e) => e.isIntersecting)) return;

210+ near.disconnect();

211+ this.loadLight()

212+ .then((light) => this.showSpectrogram(light.audio, light.fs))

213+ .catch((e) => this.abort.signal.aborted || this.fail(`Could not load the result: ${(e as Error).message}`));

214+ },

215+ { rootMargin: '400px' },

216+ );

217+ near.observe(this);

218+ this.cleanup.push(() => near.disconnect());

197219 const soundBtn = this.q<HTMLButtonElement>('sound-btn');

198220 soundBtn.hidden = false;

199221 soundBtn.addEventListener('click', async () => {

@@ -270,6 +292,7 @@ class ToyInstrumentSim extends HTMLElement {

270292 const r = await running.promise;

271293 runBtn.hidden = true;

272294 this.setAudio(r.audio, r.fs);

295+ this.showSpectrogram(r.audio, r.fs);

273296 this.showMotion(r);

274297 const ran = `Simulated in this browser with ${r.backend ?? 'CPU'}${r.fallback ? ` (WebGPU failed: ${r.fallback})` : ''} in ${duration(r.wallSeconds)}.`;

275298 if (!getPasscode()) return this.status(`${ran} Not stored: this browser has no cache passcode.`);

@@ -293,6 +316,27 @@ class ToyInstrumentSim extends HTMLElement {

293316 }

294317 }

295318

319+ // the spectrogram, with a choice of its top frequency (attribute fmax, default 5000 Hz)

320+ private mountSpectrogram() {

321+ this.spec = new Spectrogram(this.q('spec'));

322+ const sel = this.q<HTMLSelectElement>('fmax');

323+ const attr = Number(this.getAttribute('fmax'));

324+ const fmax = attr > 0 ? attr : 5000;

325+ const choices = [...new Set([...FMAX_CHOICES, fmax])].sort((a, b) => a - b);

326+ sel.innerHTML = choices.map((v) => `<option value="${v}">${v}</option>`).join('');

327+ sel.value = String(fmax);

328+ this.spec.fmax = fmax;

329+ sel.addEventListener('change', () => {

330+ this.spec.fmax = Number(sel.value);

331+ this.spec.draw();

332+ });

333+ }

334+

335+ private showSpectrogram(audio: Float32Array, fs: number) {

336+ this.q('spec-block').hidden = false;

337+ this.spec.setSignal(audio, fs);

338+ }

339+

296340 private showMotion(r: SimResult) {

297341 this.timeline.setTimes(r.frameTimes);

298342 this.scene.showReplay({ ...this.built.scene, frames: r.frames, malletPos: r.malletPos });

@@ -393,6 +437,7 @@ class ToyInstrumentSim extends HTMLElement {

393437 const dark = scheme.includes('dark') && (!scheme.includes('light') || media.matches);

394438 box.classList.toggle('dark', dark);

395439 this.scene.restyle();

440+ this.spec.draw();

396441 };

397442 apply();

398443 const obs = new MutationObserver(apply);

@@ -432,6 +477,9 @@ class ToyInstrumentSim extends HTMLElement {

432477 }

433478 }

434479

480+// choices for the spectrogram's top frequency (Hz)

481+const FMAX_CHOICES = [1000, 2500, 5000, 10000, 24000];

482+

435483 // choices in the motion bar's exaggeration dropdown

436484 const EXAGGERATIONS = [1, 2, 5, 10, 20, 50, 100];

437485

src/ui/plot.tsmodified+5−5View file

@@ -76,8 +76,8 @@ export function drawAxes(

7676 const Y = (v: number) => h - PAD.b - ((v - yr[0]) / (yr[1] - yr[0])) * (h - PAD.t - PAD.b);

7777 ctx.font = '11px system-ui, sans-serif';

7878 ctx.lineWidth = 1;

79- ctx.strokeStyle = css('--grid');

80- ctx.fillStyle = css('--text-muted');

79+ ctx.strokeStyle = css('--grid', ctx.canvas);

80+ ctx.fillStyle = css('--text-muted', ctx.canvas);

8181 ctx.textAlign = 'right';

8282 ctx.textBaseline = 'middle';

8383 for (const t of niceTicks(yr[0], yr[1], 4)) {

@@ -95,12 +95,12 @@ export function drawAxes(

9595 for (const t of niceTicks(xr[0], xr[1], Math.max(3, Math.floor((w - PAD.l) / 90)))) {

9696 ctx.fillText(fx(t), X(t), h - PAD.b + 5);

9797 }

98- ctx.strokeStyle = css('--axis');

98+ ctx.strokeStyle = css('--axis', ctx.canvas);

9999 ctx.beginPath();

100100 ctx.moveTo(PAD.l, h - PAD.b + 0.5);

101101 ctx.lineTo(w - PAD.r, h - PAD.b + 0.5);

102102 ctx.stroke();

103- ctx.fillStyle = css('--text-secondary');

103+ ctx.fillStyle = css('--text-secondary', ctx.canvas);

104104 ctx.fillText(o.xLabel, PAD.l + (w - PAD.l - PAD.r) / 2, h - 14);

105105 ctx.save();

106106 ctx.translate(12, PAD.t + (h - PAD.t - PAD.b) / 2);

@@ -237,7 +237,7 @@ export class LinePlot {

237237

238238 if (this.hoverX !== null && this.hoverX >= PAD.l && this.hoverX <= w - PAD.r) {

239239 const xv = xr[0] + ((this.hoverX - PAD.l) / plotW) * (xr[1] - xr[0]);

240- ctx.strokeStyle = css('--axis');

240+ ctx.strokeStyle = css('--axis', ctx.canvas);

241241 ctx.lineWidth = 1;

242242 ctx.beginPath();

243243 ctx.moveTo(Math.round(this.hoverX) + 0.5, PAD.t);

src/ui/spectrogram.tsmodified+4−4View file

@@ -6,9 +6,9 @@ function hexToRgb(h: string): [number, number, number] {

66 return [(v >> 16) & 255, (v >> 8) & 255, v & 255];

77 }

88

9-// Sequential ramp read from a CSS custom property: comma-separated hex steps.

10-export function rampLUT(varName: string): Uint8ClampedArray {

11- const stops = css(varName).split(',').map((s) => hexToRgb(s.trim()));

9+// Sequential ramp read from a CSS custom property (as seen by el): comma-separated hex steps.

10+export function rampLUT(varName: string, el?: Element): Uint8ClampedArray {

11+ const stops = css(varName, el).split(',').map((s) => hexToRgb(s.trim()));

1212 const lut = new Uint8ClampedArray(256 * 3);

1313 for (let i = 0; i < 256; i++) {

1414 const t = (i / 255) * (stops.length - 1);

@@ -50,7 +50,7 @@ export class Spectrogram {

5050 const nf = s.mags.length;

5151 let top = -Infinity;

5252 for (const m of s.mags) for (let k = 0; k <= nb; k++) top = Math.max(top, m[k]);

53- const lut = rampLUT('--ramp');

53+ const lut = rampLUT('--ramp', this.canvas);

5454 const img = new ImageData(nf, nb);

5555 for (let j = 0; j < nf; j++) {

5656 const m = s.mags[j];