Add scripts/compare-perf.mjs, and rule out CPU command encoding - concept-collection/turing-sphere-2

concept-collection / turing-sphere-2

Add scripts/compare-perf.mjs, and rule out CPU command encoding

A step is ~47 WebGPU calls, so command encoding was the obvious next suspect for the terminal still outrunning the browser. Measured, it goes the other way: 0.009 ms/step in Chrome against 0.062 ms/step under node-webgpu, because Chrome defers commands to the GPU process while node-webgpu validates them inline. Encoding is cheaper in the browser, so it cannot be the cause. Both the benchmark and the new script report it so this stays checkable rather than becoming folklore. compare-perf.mjs measures the same solver work in both environments — same .m, same kernels, batched, nothing read back, and no renderer on either side, since the browser side runs test.html?soak= — and reports each with its encoding share, Fourier stage and adapter. If they agree, the solver is equally fast in the browser and everything the app shows on top is readback, rendering and animation pacing; if the browser is slower at this, it is the GPU stack itself. It refuses to interpret the ratio when the two are not the same device. A browser falling back to a software adapter is a common cause of "the browser is much slower", and it makes the comparison meaningless rather than informative — so that case is called out with a pointer to chrome://gpu instead of being reported as a number. The soak now also posts its measurements structurally for that script, and splits its own solver rate into encoding and GPU time.

Jeremy Magland <jmagland@flatironinstitute.org> committed 20 days ago commit 17db8f17874b parent cde22ea Browse files

4 changed files+240−1

README.mdmodified+22−0View file

@@ -230,6 +230,19 @@ Other things the comparison does not control for:

230230 Some gap is real and some is measurement. Four numbers, in increasing order of

231231 what they include — walk down them and the gap attributes itself:

232232

233+```

234+node scripts/compare-perf.mjs [--lmax 63] [--steps 300]

235+```

236+

237+measures the same solver work in both — batched, nothing read back, no rendering

238+on either side — and reports each with its CPU-encoding share, the Fourier stage,

239+and the adapter. It stops you first if the two are not even the same device: a

240+browser quietly falling back to a software adapter is a common cause of "the

241+browser is much slower", and then the ratio compares different hardware and means

242+nothing.

243+

244+By hand, four numbers, in increasing order of what they include:

245+

233246 | number | includes |

234247 |---|---|

235248 | `npm run bench -- --lmax 63` | desktop solver: batched steps, one sync per batch, in-process Dawn |

@@ -246,6 +259,12 @@ remaining suspects are:

246259 most when the GPU is fast. `npm run bench -- --batch 4` makes the desktop pay a

247260 sync as often as the app's frame loop does, which shows how much of the gap is

248261 just amortization.

262+- **not CPU command encoding**, which is worth ruling out explicitly because it is

263+ the obvious suspect: a step is ~47 WebGPU calls, and 32 of them per burst is a

264+ lot of JS→GPU traffic. Measured, it goes the other way — 0.009 ms/step in Chrome

265+ against 0.062 ms/step under node-webgpu, because Chrome defers commands to the

266+ GPU process while node-webgpu validates them inline. Encoding is *cheaper* in

267+ the browser. Both `compare-perf.mjs` and the benchmark print it.

249268 - **competing with the renderer.** The page draws two spheres through WebGL on the

250269 same GPU, in its own animation loop. The soak has no renderer, so comparing the

251270 soak against the app's `solver` separates contention from everything else.

@@ -342,6 +361,9 @@ Other commands:

342361 - `node scripts/compare-env.mjs` — run one identical spec on the desktop and in a

343362 browser and compare the final state (see

344363 [Is it really the same computation?](#is-it-really-the-same-computation)).

364+- `node scripts/compare-perf.mjs` — measure the same solver work in both and split

365+ the difference (see

366+ [Why the browser is slower](#why-the-browser-is-slower-and-how-to-find-out-by-how-much)).

345367 - `test.html?soak=<steps>&lmax=<n>` — solver-only soak with no rendering.

346368

347369 ### A note on canvas resizing

scripts/bench.tsmodified+14−1View file

@@ -196,9 +196,12 @@ try {

196196 let lastReport = performance.now();

197197 const tp0 = performance.now();

198198 let stepsRun = 0;

199+ let encodeMs = 0;

199200 for (let b = 0; b < batches; b++) {

200201 const n = Math.min(batch, spec.steps - stepsRun);

202+ const e0 = performance.now();

201203 session.step(n);

204+ encodeMs += performance.now() - e0;

202205 await done();

203206 stepsRun += n;

204207 if (progress && performance.now() - lastReport > 1000) {

@@ -210,6 +213,7 @@ try {

210213 }

211214 }

212215 const throughputMs = (performance.now() - tp0) / stepsRun;

216+ const encodePerStep = encodeMs / stepsRun;

213217 if (progress) process.stderr.write('\r\x1b[K');

214218

215219 // --- latency: one step per submit, for the distribution ---

@@ -252,7 +256,12 @@ try {

252256 grid: { lmax: cfg.lmax, nlat: cfg.nlat, nphi: cfg.nphi, nlm: session.sht.nlm },

253257 compiled: { opsPerStep: plan.step.length, kernels },

254258 digest,

255- throughput: { batch, msPerStep: throughputMs, stepsPerSec: 1000 / throughputMs },

259+ throughput: {

260+ batch,

261+ msPerStep: throughputMs,

262+ stepsPerSec: 1000 / throughputMs,

263+ encodeMsPerStep: encodePerStep,

264+ },

256265 latency: t,

257266 state: {

258267 t: session.t,

@@ -274,6 +283,10 @@ try {

274283 `${(spec.params.dt * (1000 / throughputMs)).toFixed(2)} model time/s` +

275284 ` (batches of ${batch})`,

276285 );

286+ console.log(

287+ ` of which CPU command encoding: ${encodePerStep.toFixed(3)} ms/step ` +

288+ `(${((100 * encodePerStep) / throughputMs).toFixed(0)}% — the rest is the GPU)`,

289+ );

277290 console.log(

278291 ` one step per submit: ${t.meanMs.toFixed(2)} ms mean · median ${t.medianMs.toFixed(2)} · ` +

279292 `p05 ${t.p05Ms.toFixed(2)} · p95 ${t.p95Ms.toFixed(2)} · min ${t.minMs.toFixed(2)}`,

scripts/compare-perf.mjsadded+177−0View file

@@ -0,0 +1,177 @@

1+/**

2+ * Why is the terminal faster than the browser?

3+ *

4+ * Measures the *same* solver work — same .m, same kernels, batched, nothing read

5+ * back, no rendering on either side — in the terminal (Dawn, in-process) and in a

6+ * real browser, and splits the result so the gap attributes itself:

7+ *

8+ * node scripts/compare-perf.mjs [--lmax 63] [--steps 300] [--preset schnak-spots]

9+ *

10+ * The browser side runs `test.html?soak=`, which has no renderer at all. So:

11+ *

12+ * - if the two agree, the solver is equally fast in the browser, and whatever

13+ * the app shows on top of this is readback, rendering, and animation pacing.

14+ * - if the browser is slower here, it is the GPU stack itself: submits crossing

15+ * into the GPU process, or Metal/Vulkan execution differing between Chrome's

16+ * Dawn and node-webgpu's.

17+ *

18+ * CPU command encoding is reported for both, because it is the one cost that can

19+ * make a fast GPU irrelevant — and it is usually *cheaper* in the browser, which

20+ * defers commands to the GPU process instead of validating them inline.

21+ *

22+ * Requires `npm run build` first, and desktop WebGPU for the terminal side.

23+ */

24+import { createServer } from 'node:http';

25+import { readFile } from 'node:fs/promises';

26+import { extname, join } from 'node:path';

27+import { spawnSync } from 'node:child_process';

28+import puppeteer from 'puppeteer-core';

29+

30+const argv = process.argv.slice(2);

31+const flag = (name, dflt) => {

32+ const i = argv.indexOf(`--${name}`);

33+ if (i >= 0 && argv[i + 1] !== undefined) return argv[i + 1];

34+ const eq = argv.find((a) => a.startsWith(`--${name}=`));

35+ return eq ? eq.slice(name.length + 3) : dflt;

36+};

37+const lmax = flag('lmax', '63');

38+const steps = flag('steps', '300');

39+const preset = flag('preset', 'schnak-spots');

40+

41+console.log(`comparing solver rate — preset ${preset}, lmax ${lmax}, ${steps} steps\n`);

42+

43+// ---- terminal ------------------------------------------------------------

44+const bench = spawnSync(

45+ 'npx',

46+ [

47+ 'vite-node', 'scripts/bench.ts', '--json',

48+ '--preset', preset, '--lmax', lmax, '--steps', steps, '--warmup', '30',

49+ ],

50+ { encoding: 'utf8', maxBuffer: 64 * 1024 * 1024 },

51+);

52+if (bench.status !== 0) {

53+ console.error(bench.stdout ?? '');

54+ console.error(bench.stderr ?? '');

55+ console.error('compare-perf: the terminal run failed');

56+ process.exit(1);

57+}

58+const desktop = JSON.parse(bench.stdout);

59+

60+// ---- browser -------------------------------------------------------------

61+const DIST = new URL('../dist/', import.meta.url).pathname;

62+const MIME = { '.html': 'text/html', '.js': 'text/javascript', '.css': 'text/css' };

63+const server = createServer(async (req, res) => {

64+ try {

65+ const path = req.url === '/' ? '/index.html' : req.url.split('?')[0];

66+ const data = await readFile(join(DIST, path));

67+ res.writeHead(200, {

68+ 'content-type': MIME[extname(path)] ?? 'application/octet-stream',

69+ });

70+ res.end(data);

71+ } catch {

72+ res.writeHead(404);

73+ res.end('not found');

74+ }

75+});

76+await new Promise((r) => server.listen(0, '127.0.0.1', r));

77+const port = server.address().port;

78+

79+const flagSets = [

80+ ['--headless=new', '--no-sandbox', '--enable-unsafe-webgpu', '--enable-features=Vulkan'],

81+ ['--headless=new', '--no-sandbox', '--enable-unsafe-webgpu',

82+ '--use-webgpu-adapter=swiftshader', '--enable-unsafe-swiftshader'],

83+];

84+

85+let soak = null;

86+let lastError = '';

87+for (const args of flagSets) {

88+ let browser;

89+ try {

90+ browser = await puppeteer.launch({

91+ executablePath: process.env.CHROME_PATH ?? '/usr/bin/google-chrome',

92+ args,

93+ });

94+ const page = await browser.newPage();

95+ page.on('pageerror', (e) => (lastError = e.message));

96+ await page.goto(`http://127.0.0.1:${port}/test.html?soak=${steps}&lmax=${lmax}`, {

97+ waitUntil: 'load',

98+ });

99+ await page.waitForFunction(() => window.__SOAK__ !== undefined, { timeout: 600000 });

100+ soak = await page.evaluate(() => window.__SOAK__);

101+ await browser.close();

102+ break;

103+ } catch (e) {

104+ lastError = e.message ?? String(e);

105+ await browser?.close();

106+ }

107+}

108+server.close();

109+

110+if (!soak) {

111+ console.error(`compare-perf: the browser run failed: ${lastError}`);

112+ process.exit(1);

113+}

114+

115+// ---- report --------------------------------------------------------------

116+const d = desktop.throughput;

117+const row = (label, total, encode, adapter, fourier) => {

118+ console.log(` ${label.padEnd(10)} ${total.toFixed(3)} ms/step` +

119+ ` encoding ${encode.toFixed(3)} ms/step (${((100 * encode) / total).toFixed(0)}%)` +

120+ ` ${fourier.toUpperCase()}`);

121+ console.log(` ${''.padEnd(10)} ${adapter}`);

122+};

123+console.log('solver only, batched, nothing read back, no rendering:\n');

124+row('terminal', d.msPerStep, d.encodeMsPerStep, desktop.backend.adapter, desktop.digest?.fourier ?? 'fft');

125+row('browser', soak.solverMsPerStep, soak.encodeMsPerStep, soak.adapter, soak.fourier);

126+

127+const ratio = soak.solverMsPerStep / d.msPerStep;

128+console.log(`\n browser / terminal = ${ratio.toFixed(2)}x`);

129+

130+// Before reading anything into the ratio: are these even the same GPU? A browser

131+// quietly falling back to a software adapter is a common cause of "the browser is

132+// much slower", and it makes the comparison meaningless rather than informative.

133+const software = (a) => /swiftshader|llvmpipe|software|basic render/i.test(a ?? '');

134+const desktopAdapter = desktop.backend.adapter ?? '';

135+if (software(soak.adapter) !== software(desktopAdapter)) {

136+ console.log(

137+ `\n STOP these are not the same device. One side is a software renderer:\n` +

138+ ` terminal: ${desktopAdapter}\n browser: ${soak.adapter}\n` +

139+ ` The ratio above compares different hardware and means nothing. If it is the\n` +

140+ ` browser that fell back, that IS the answer — check chrome://gpu for why\n` +

141+ ` (hardware acceleration disabled, or the GPU blocklisted).`,

142+ );

143+} else if (desktopAdapter && soak.adapter && desktopAdapter !== soak.adapter) {

144+ console.log(

145+ `\n NOTE the two report different adapters, which may just be different\n` +

146+ ` naming for the same GPU — but check it is not a second GPU:\n` +

147+ ` terminal: ${desktopAdapter}\n browser: ${soak.adapter}`,

148+ );

149+}

150+

151+if (desktop.digest && desktop.digest.fourier !== soak.fourier) {

152+ console.log(

153+ `\n NOTE different Fourier stage (${desktop.digest.fourier} vs ${soak.fourier}).\n` +

154+ ` Those are different algorithms with different cost — that is the difference,\n` +

155+ ` not a symptom of it.`,

156+ );

157+} else if (ratio < 1.3) {

158+ console.log(

159+ `\n The solver runs at the same rate in both. Anything the app shows beyond\n` +

160+ ` this is its readback per species, the colormapping, competing with the\n` +

161+ ` renderer for the GPU, and animation pacing — not the computation.`,

162+ );

163+} else {

164+ console.log(

165+ `\n The browser is slower at the same solver work, with no renderer involved,\n` +

166+ ` so it is the GPU stack rather than anything above it: every submit crosses\n` +

167+ ` into the GPU process, and Chrome's Dawn and node-webgpu's need not compile\n` +

168+ ` or schedule these shaders identically. Note also that an animation-paced\n` +

169+ ` page can leave the GPU in a low-power state where a continuous benchmark\n` +

170+ ` boosts it; this soak hammers it continuously, so if the app is slower than\n` +

171+ ` this number, that is a likely reason.`,

172+ );

173+}

174+console.log(

175+ `\n Correctness is a separate question: scripts/compare-env.mjs checks that the\n` +

176+ ` two environments compute the same state.`,

177+);

test/test-page.tsmodified+27−0View file

@@ -21,6 +21,16 @@ declare global {

2121 __RESULTS__?: { ok: boolean; fatal?: string; lines: string[] };

2222 /** Set by the ?state= mode, for scripts/compare-env.mjs. */

2323 __STATE__?: { digest: StateDigest; state: number[] };

24+ /** Set by the ?soak= mode, for scripts/compare-perf.mjs. */

25+ __SOAK__?: {

26+ lmax: number;

27+ steps: number;

28+ batch: number;

29+ solverMsPerStep: number;

30+ encodeMsPerStep: number;

31+ adapter: string;

32+ fourier: 'fft' | 'dft';

33+ };

2434 }

2535 }

2636

@@ -67,11 +77,17 @@ async function soak(steps: number, lmax: number): Promise<void> {

6777 // waited for, never read back, so it is comparable to `npm run bench`.

6878 let solverMs = 0;

6979 let solverSteps = 0;

80+ let encodeMs = 0;

7081 const t0 = performance.now();

7182 for (let s = 0; s < steps; s += BATCH) {

7283 const n = Math.min(BATCH, steps - s);

7384 const b0 = performance.now();

7485 session.step(n);

86+ // CPU-side command encoding, separated from GPU execution: in a browser each

87+ // WebGPU call crosses Blink's bindings and Dawn's validation, so on a fast

88+ // GPU the encoding can be what actually limits the step rate.

89+ const b1 = performance.now();

90+ encodeMs += b1 - b0;

7591 await session.sync();

7692 solverMs += performance.now() - b0;

7793 solverSteps += n;

@@ -99,10 +115,12 @@ async function soak(steps: number, lmax: number): Promise<void> {

99115 let finite = true;

100116 for (const v of final) if (!Number.isFinite(v)) finite = false;

101117 const solverPerStep = solverMs / solverSteps;

118+ const encodePerStep = encodeMs / solverSteps;

102119 check(

103120 `soak: ${steps} steps survived`,

104121 finite,

105122 `solver ${solverPerStep.toFixed(2)} ms/step (batches of ${BATCH}, no readback), ` +

123+ `of which ${encodePerStep.toFixed(3)} ms/step CPU encoding, ` +

106124 `${ms.toFixed(2)} ms/step incl. sampling readback`,

107125 );

108126 log(

@@ -110,6 +128,15 @@ async function soak(steps: number, lmax: number): Promise<void> {

110128 ` same .m, same kernels, no rendering on either side.`,

111129 );

112130

131+ window.__SOAK__ = {

132+ lmax,

133+ steps,

134+ batch: BATCH,

135+ solverMsPerStep: solverPerStep,

136+ encodeMsPerStep: encodePerStep,

137+ adapter: await describeAdapter(device),

138+ fourier: session.sht.fourierMode,

139+ };

113140 session.destroy();

114141 window.__RESULTS__ = { ok: failures === 0, lines };

115142 log(failures === 0 ? 'ALL PASS' : `${failures} FAILURE(S)`);