Fill the cache from the command line, without a browser
TURING_SURFACE_CACHE_KEY=... npx https://.../fill.tgz
npm installs a tarball from a URL as happily as from a package name, so
nothing goes to the registry: the deploy workflow builds the bundle and packs
it beside the page, which keeps the command line at the same commit as the app
it fills the cache for. The published manifest depends on neither numbl nor a
checkout -- the compiler is bundled in, as it is for the page -- and its only
real dependency is h5wasm, whose node build reads its wasm off disk.
The walk, the runs and the uploads are the page's, unchanged. What node lacks
is WebGPU, so the command brings its own: the optional `webgpu` package of
prebuilt Dawn binaries, installed under the globals the transform code
expects, as turing-surface's command-line scripts already do. Dawn reaches the
GPU through Vulkan or Metal, so a machine without one falls back to a software
rasterizer that is fast enough to look like it is working and slow enough to
be worth nothing; the adapter and the steps/s are printed, and both a
software adapter and a rate below 20 steps/s are called out. It still runs --
that judgment belongs to whoever started it.
The key comes from --key, the environment, or a file `login` writes; the
environment is what the page will hand out, since argv is visible to every
user on the machine through `ps`. --dry-run lists what a machine would take
on, --limit and --model narrow it, and ctrl-C stops after the current run.
Two things move so that the two front ends cannot drift: the target-to-spec
rule (the pinned seed, the longest end time) now lives with the walk, and the
page sets its dropdowns from it; and asking whether a key may upload at all is
one request in the client, which `login` uses to reject a typo on the spot.
15 changed files+740−22
.gitignoremodified+1−0View file
@@ -2,3 +2,4 @@ node_modules/
22 dist/
33 *.log
44 *.png
5+dist-cli/
README.mdmodified+58−0View file
@@ -127,6 +127,54 @@ Worker: the client presents the API key and a file name, receives a presigned
127127 R2 PUT URL, and uploads directly. Only holders of the key can write; everyone
128128 can read. The key is entered in the page and kept in localStorage.
129129
130+## Filling it from the command line
131+
132+A browser window is a poor place to leave a long computation, so the same walk
133+runs outside one:
134+
135+```
136+TURING_SURFACE_CACHE_KEY=… npx https://concept-collection.github.io/turing-surface-cache/fill.tgz
137+```
138+
139+Nothing is published to the npm registry — npm installs a tarball from a URL
140+as happily as from a package name, and the tarball is built and deployed
141+beside the page, so the command line is always the same commit as the app.
142+The page itself offers this command, ready to copy, once an upload key is
143+entered. The key can also be saved for later runs (`login` prompts for it and
144+writes `~/.config/turing-surface-cache/key`), or passed as `--key`, though the
145+environment is preferable: a key on the command line is visible to every user
146+on the machine through `ps`, while another process's environment is not.
147+
148+The walk, the runs and the uploads are the page's own — the same modules under
149+[`src/cache/`](src/cache/), driven by console output instead of a status bar
150+(see [`src/cli/fill.ts`](src/cli/fill.ts)). What differs is the WebGPU: node
151+has none, so the command line brings its own, the optional `webgpu` package of
152+prebuilt [Google Dawn](https://dawn.googlesource.com/dawn) binaries, installed
153+under the globals the transform code expects. Dawn reaches the GPU through
154+Vulkan on Linux and Windows and Metal on macOS, so a machine wanting to
155+contribute needs a GPU and its driver — on a machine without one, Dawn
156+either finds no adapter at all or falls back to a software rasterizer, which
157+is roughly a thousand times slower and worth nothing to anybody. The command
158+names its adapter on startup, reports its rate in steps per second, and says
159+so plainly when either looks wrong; it does not refuse to run, since the
160+judgment is the operator's.
161+
162+Progress is a line per target and a rate that updates in place:
163+
164+```
165+[2] schnakenberg a=0.15 b=0.9 D1=4e-4 D2=8e-3 dt=0.05 · sphere · 2 knobs from the defaults
166+ computing to t = 1600 (32,000 steps)
167+ t = 812.4 / 1600 51% 184 steps/s eta 1m11s uploaded 3/3
168+ computed in 2m54s — uploaded 5 solutions (t = 100, 200, 400, 800, 1600)
169+```
170+
171+When the output is not a terminal the same lines are written periodically
172+instead of in place, so a `nohup`ed log stays readable. `--dry-run` lists the
173+first targets and whether each is already cached, which is a cheap way to see
174+what a machine would take on before committing it; `--limit` and `--model`
175+narrow the work; and ctrl-C stops after the current run, so nothing in flight
176+is lost.
177+
130178 ## The cache file
131179
132180 Cache files are HDF5, written in the browser with
@@ -166,8 +214,18 @@ computed a combination first, and the file records which adapter that was.
166214 npm install
167215 npm run dev # local dev server
168216 npm run build # type-check + production build to dist/
217+npm run build:cli # the command-line bundle, packed as dist/fill.tgz
169218 ```
170219
220+`npm run build` runs `build:cli` too, so a deployment carries both. The
221+command-line bundle is an SSR vite build of
222+[`src/cli/fill.ts`](src/cli/fill.ts) with everything under `src/` and numbl's
223+compiler bundled in, exactly as the page's build has them; the only things
224+left external are h5wasm, whose node build reads its wasm off disk, and Dawn,
225+which is a native addon. [`scripts/pack-cli.mjs`](scripts/pack-cli.mjs) writes
226+the published manifest, which therefore depends on neither numbl nor a
227+checkout of anything.
228+
171229 numbl is a local `file:../../numbl` dependency, exactly as in turing-surface —
172230 a sibling checkout of [numbl](https://github.com/flatironinstitute/numbl) is
173231 required, reached through the `numbl-src` alias in
package-lock.jsonmodified+18−3View file
@@ -23,6 +23,9 @@
2323 },
2424 "engines": {
2525 "node": ">=22.6"
26+ },
27+ "optionalDependencies": {
28+ "webgpu": "^0.4.0"
2629 }
2730 },
2831 "../../numbl": {
@@ -947,7 +950,7 @@
947950 "version": "0.1.71",
948951 "resolved": "https://registry.npmjs.org/@webgpu/types/-/types-0.1.71.tgz",
949952 "integrity": "sha512-mMy8/ODcKhab808co15eW+yN+HgXoQxRQHTiBV9Mrvl1r0ufnid7YOcI+gi4eUWSWl9ezD6TW2KXccrL8HCh2A==",
950- "dev": true,
953+ "devOptional": true,
951954 "license": "BSD-3-Clause"
952955 },
953956 "node_modules/agent-base": {
@@ -1228,7 +1231,7 @@
12281231 "version": "4.4.3",
12291232 "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz",
12301233 "integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==",
1231- "dev": true,
1234+ "devOptional": true,
12321235 "license": "MIT",
12331236 "dependencies": {
12341237 "ms": "^2.1.3"
@@ -1600,7 +1603,7 @@
16001603 "version": "2.1.3",
16011604 "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz",
16021605 "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==",
1603- "dev": true,
1606+ "devOptional": true,
16041607 "license": "MIT"
16051608 },
16061609 "node_modules/nanoid": {
@@ -2126,6 +2129,18 @@
21262129 }
21272130 }
21282131 },
2132+ "node_modules/webgpu": {
2133+ "version": "0.4.0",
2134+ "resolved": "https://registry.npmjs.org/webgpu/-/webgpu-0.4.0.tgz",
2135+ "integrity": "sha512-F5pimn3Aoi0zWjuRdiVs5TnrUwSzD2lESBohsIUsqyitWkGRQlXU2fhV6ycXlQTa1bvAf3sjqiUpBEpmSQ5ptA==",
2136+ "hasInstallScript": true,
2137+ "license": "MIT",
2138+ "optional": true,
2139+ "dependencies": {
2140+ "@webgpu/types": "^0.1.69",
2141+ "debug": "^4.4.0"
2142+ }
2143+ },
21292144 "node_modules/wrap-ansi": {
21302145 "version": "7.0.0",
21312146 "resolved": "https://registry.npmjs.org/wrap-ansi/-/wrap-ansi-7.0.0.tgz",
package.jsonmodified+5−1View file
@@ -9,13 +9,17 @@
99 "license": "CECILL-2.1",
1010 "scripts": {
1111 "dev": "vite",
12- "build": "tsc --noEmit && vite build"
12+ "build": "tsc --noEmit && vite build && npm run build:cli",
13+ "build:cli": "vite build --config vite.cli.config.ts && node scripts/pack-cli.mjs"
1314 },
1415 "dependencies": {
1516 "h5wasm": "^0.10.3",
1617 "numbl": "file:../../numbl",
1718 "three": "^0.183.0"
1819 },
20+ "optionalDependencies": {
21+ "webgpu": "^0.4.0"
22+ },
1923 "devDependencies": {
2024 "@types/node": "^26.1.1",
2125 "@types/three": "^0.185.1",
scripts/pack-cli.mjsadded+55−0View file
@@ -0,0 +1,55 @@
1+/**
2+ * Pack the command-line bundle as an npm tarball and put it where the page is
3+ * deployed, so that
4+ *
5+ * npx https://concept-collection.github.io/turing-surface-cache/fill.tgz
6+ *
7+ * installs and runs it anywhere node is. Nothing is published to the registry:
8+ * npm installs a tarball from a URL as happily as from a package name, and
9+ * this way the command line is always the same commit as the page it fills the
10+ * cache for.
11+ *
12+ * The package.json written here is the whole published manifest. It has no
13+ * numbl dependency — the compiler is bundled into fill.js exactly as it is
14+ * into the page — and its only real dependency is h5wasm, whose node build
15+ * reads its wasm off disk and so cannot be bundled. Dawn is optional, so the
16+ * install still succeeds on a machine that has no prebuilt binary for it; the
17+ * CLI says what to do about that if it comes to it.
18+ *
19+ * Usage: node scripts/pack-cli.mjs (after `vite build --config vite.cli.config.ts`)
20+ */
21+import { execFileSync } from 'node:child_process';
22+import { mkdtemp, copyFile, writeFile, readFile, rm, mkdir } from 'node:fs/promises';
23+import { tmpdir } from 'node:os';
24+import { join } from 'node:path';
25+
26+const root = new URL('..', import.meta.url).pathname;
27+const pkg = JSON.parse(await readFile(join(root, 'package.json'), 'utf8'));
28+
29+const manifest = {
30+ name: 'turing-surface-fill',
31+ version: pkg.version,
32+ description: 'Fill the turing-surface-cache shared cache from the command line',
33+ license: pkg.license,
34+ type: 'module',
35+ engines: pkg.engines,
36+ bin: { 'turing-surface-fill': 'fill.js' },
37+ files: ['fill.js'],
38+ dependencies: { h5wasm: pkg.dependencies.h5wasm },
39+ optionalDependencies: { webgpu: pkg.optionalDependencies.webgpu },
40+};
41+
42+const stage = await mkdtemp(join(tmpdir(), 'turing-fill-'));
43+try {
44+ await copyFile(join(root, 'dist-cli/fill.js'), join(stage, 'fill.js'));
45+ await writeFile(join(stage, 'package.json'), `${JSON.stringify(manifest, null, 2)}\n`);
46+ const out = execFileSync('npm', ['pack', '--silent', '--pack-destination', stage], {
47+ cwd: stage,
48+ encoding: 'utf8',
49+ }).trim();
50+ await mkdir(join(root, 'dist'), { recursive: true });
51+ await copyFile(join(stage, out), join(root, 'dist/fill.tgz'));
52+ console.log(`dist/fill.tgz (${manifest.name} ${manifest.version})`);
53+} finally {
54+ await rm(stage, { recursive: true, force: true });
55+}
src/cache/autoWalk.tsmodified+28−0View file
@@ -26,9 +26,15 @@ import {
2626 DEFAULT_MODEL_KEY,
2727 GEOMETRY_CHOICES,
2828 AUTO_DT,
29+ AUTO_SEED,
30+ T_END_CHOICE,
31+ LMAX,
32+ NITER,
33+ LAM3,
2934 type DiscreteChoice,
3035 } from './options.ts';
3136 import { DEFAULT_GEOMETRY_KEY } from '../geom/registry.ts';
37+import { APP_NAME, FORMAT_VERSION, type CacheSpec } from './spec.ts';
3238
3339 export interface AutoTarget {
3440 model: string;
@@ -108,6 +114,28 @@ export function enumerateTargets(): AutoTarget[] {
108114 return out;
109115 }
110116
117+/**
118+ * The solution a target names. The seed is the pinned one, and the end time is
119+ * the longest listed: a run reaching it passes through every shorter one and
120+ * contributes those on the way, so one run fills the whole chain. The page
121+ * sets its dropdowns from this rather than deciding the same thing twice.
122+ */
123+export function specForTarget(target: AutoTarget): CacheSpec {
124+ return {
125+ app: APP_NAME,
126+ formatVersion: FORMAT_VERSION,
127+ model: target.model,
128+ params: { ...target.params },
129+ geometry: target.geometry,
130+ geometryParams: { ...target.geometryParams },
131+ lmax: LMAX,
132+ niter: NITER,
133+ lam3: LAM3,
134+ seed: AUTO_SEED,
135+ tEnd: Math.max(...T_END_CHOICE.values),
136+ };
137+}
138+
111139 /**
112140 * The walk order: by distance, randomly within each distance. Shuffling the
113141 * whole list and then sorting by distance gives exactly that, since Array's
src/cache/client.tsmodified+26−8View file
@@ -10,7 +10,7 @@
1010 * receive a presigned R2 PUT URL, and upload directly. Only holders of the
1111 * key can write; everyone can read.
1212 */
13-import { cacheFileName, canonicalJson, type CacheSpec } from './spec.ts';
13+import { APP_NAME, FORMAT_VERSION, cacheFileName, canonicalJson, type CacheSpec } from './spec.ts';
1414
1515 const PUBLIC_BASE = 'https://tempory.net/tmpbucket/';
1616 const WORKER_BASE = 'https://tmpbucket.figurl.workers.dev';
@@ -57,13 +57,9 @@ export async function isCached(lookup: CacheLookup): Promise<boolean> {
5757 return (await headCached(lookup)) === true;
5858 }
5959
60-/** Upload one cache file. Resolves to its public URL. */
61-export async function uploadCacheFile(
62- apiKey: string,
63- fileName: string,
64- bytes: Uint8Array,
65-): Promise<string> {
66- const res = await fetch(`${WORKER_BASE}/api/upload-url`, {
60+/** Ask the Worker for a presigned PUT. Nothing is written until it is used. */
61+function requestUploadUrl(apiKey: string, fileName: string): Promise<Response> {
62+ return fetch(`${WORKER_BASE}/api/upload-url`, {
6763 method: 'POST',
6864 headers: {
6965 Authorization: `Bearer ${apiKey}`,
@@ -71,6 +67,28 @@ export async function uploadCacheFile(
7167 },
7268 body: JSON.stringify({ fileName, contentType: CONTENT_TYPE }),
7369 });
70+}
71+
72+/**
73+ * Would this key be allowed to upload? Asking for a grant and not using it
74+ * writes nothing, and answers in one request — worth doing when a key is
75+ * entered, since the alternative is finding out after the first run.
76+ * Throws if the Worker cannot be reached at all, which is not the key's fault.
77+ */
78+export async function verifyApiKey(apiKey: string): Promise<boolean> {
79+ const res = await requestUploadUrl(apiKey, `${APP_NAME}/v${FORMAT_VERSION}/.keycheck`);
80+ if (res.status === 401 || res.status === 403) return false;
81+ if (!res.ok) throw new Error(`upload-url request failed: HTTP ${res.status}`);
82+ return true;
83+}
84+
85+/** Upload one cache file. Resolves to its public URL. */
86+export async function uploadCacheFile(
87+ apiKey: string,
88+ fileName: string,
89+ bytes: Uint8Array,
90+): Promise<string> {
91+ const res = await requestUploadUrl(apiKey, fileName);
7492 if (res.status === 401 || res.status === 403) {
7593 throw new Error('upload not authorized — check the API key');
7694 }
src/cache/h5file.tsmodified+7−1View file
@@ -71,7 +71,13 @@ interface EmFS {
7171 let scratchCounter = 0;
7272
7373 async function withH5<T>(fn: (h5: H5Module, fs: EmFS) => T | Promise<T>): Promise<T> {
74- const h5 = (await import('h5wasm')) as unknown as H5Module;
74+ // Two builds of the same library: the browser one carries the wasm inside
75+ // the bundle, the node one reads it off disk. __NODE_BUILD__ is a build-time
76+ // constant (see vite.config.ts and vite.cli.config.ts), so whichever branch
77+ // this build is not takes no part in it.
78+ const h5 = (await (__NODE_BUILD__
79+ ? import('h5wasm/node')
80+ : import('h5wasm'))) as unknown as H5Module;
7581 const { FS } = (await h5.ready) as { FS: EmFS };
7682 return fn(h5, FS);
7783 }
src/cli/fill.tsadded+333−0View file
@@ -0,0 +1,333 @@
1+/**
2+ * Filling the shared cache from the command line, so that a machine with a GPU
3+ * and nothing to do can contribute without a browser window open on it.
4+ *
5+ * The walk, the runs and the uploads are the page's (src/cache/fillWalk.ts,
6+ * src/cache/runSpec.ts); what is here is the shell around them — Dawn instead
7+ * of a browser's WebGPU, a key from the environment instead of localStorage,
8+ * and lines of text instead of a status bar.
9+ */
10+import { requestShtDevice, describeAdapter } from '../sht/sht.ts';
11+import { mModelByKey } from '../mgpu/registry.ts';
12+import { formatFailure } from '../mgpu/errors.ts';
13+import {
14+ fmtChoice,
15+ GEOMETRY_CHOICES,
16+ MODEL_CHOICES,
17+ T_END_CHOICE,
18+ type DiscreteChoice,
19+} from '../cache/options.ts';
20+import { autoOrder, specForTarget, type AutoTarget } from '../cache/autoWalk.ts';
21+import { headCached, lookupFor, verifyApiKey } from '../cache/client.ts';
22+import { SolverSession } from '../cache/solver.ts';
23+import { fillWalk } from '../cache/fillWalk.ts';
24+import { stepsFor } from '../cache/spec.ts';
25+import type { RunSummary } from '../cache/runSpec.ts';
26+import { installWebGpu, errMsg, isSoftwareAdapter, NO_ADAPTER_HINT } from './webgpu.ts';
27+import { KEY_ENV, keyPath, maskKey, promptSecret, resolveKey, saveKey } from './key.ts';
28+
29+const HELP = `turing-surface-cache — fill the shared cache of Turing patterns
30+
31+Usage
32+ fill [options] work through the parameter space, contributing what is
33+ missing, until stopped (ctrl-C)
34+ fill --dry-run [N] show the first N targets and whether they are cached
35+ login save an upload key for later runs
36+ --help
37+
38+Options
39+ --key <key> upload key; otherwise $${KEY_ENV}, otherwise the saved key
40+ --limit <n> stop after n solutions have been computed
41+ --model <key> only targets of one model (schnakenberg, brusselator,
42+ allencahn)
43+ --tend <list> replace the end-time list, e.g. --tend 5,10 (for testing:
44+ a short run hashes to its own honest cache entry)
45+
46+An upload key is required: the walk exists to contribute. Solutions are read
47+by everyone and written only by key holders.`;
48+
49+interface Options {
50+ command: 'fill' | 'login' | 'help';
51+ key?: string;
52+ limit: number;
53+ model?: string;
54+ dryRun: number;
55+}
56+
57+function parseArgs(argv: string[]): Options {
58+ const opts: Options = { command: 'fill', limit: Infinity, dryRun: 0 };
59+ const rest = [...argv];
60+ if (rest[0] === 'fill' || rest[0] === 'login') opts.command = rest.shift() as 'fill' | 'login';
61+ /** A count option whose value may be left off (--dry-run, --dry-run 40). */
62+ const count = (fallback: number): number => {
63+ const next = rest[0];
64+ if (next && /^\d+$/.test(next)) return Number(rest.shift());
65+ return fallback;
66+ };
67+ while (rest.length) {
68+ const arg = rest.shift()!;
69+ if (arg === '--help' || arg === '-h') opts.command = 'help';
70+ else if (arg === '--key') opts.key = rest.shift();
71+ else if (arg === '--limit') opts.limit = Number(rest.shift());
72+ else if (arg === '--model') opts.model = rest.shift();
73+ else if (arg === '--dry-run') opts.dryRun = count(20);
74+ else if (arg === '--tend') setEndTimes(rest.shift());
75+ else throw new Error(`unknown option ${arg}`);
76+ }
77+ if (opts.model && !mModelByKey(opts.model)) throw new Error(`unknown model ${opts.model}`);
78+ if (!(opts.limit > 0)) throw new Error('--limit wants a positive number');
79+ return opts;
80+}
81+
82+/** The page's ?tend hook, spelled as an option (src/main.ts). */
83+function setEndTimes(list: string | undefined): void {
84+ const values = (list ?? '')
85+ .split(',')
86+ .map(Number)
87+ .filter((v) => Number.isFinite(v) && v > 0);
88+ if (!values.length) throw new Error('--tend wants a comma-separated list of end times');
89+ T_END_CHOICE.values = values;
90+ T_END_CHOICE.value = values[0];
91+}
92+
93+// ---------------------------------------------------------------- output
94+const tty = process.stdout.isTTY === true;
95+/** Written in place on a terminal, and only every so often when piped, so a
96+ * log file does not fill with progress. */
97+const LOG_EVERY_MS = 30_000;
98+let liveLine = false;
99+
100+function say(line = ''): void {
101+ if (liveLine) {
102+ process.stdout.write('\n');
103+ liveLine = false;
104+ }
105+ process.stdout.write(`${line}\n`);
106+}
107+
108+/** One line that keeps being rewritten while a run advances. */
109+function live(line: string): void {
110+ if (!tty) return;
111+ process.stdout.write(`\r${line.padEnd(78).slice(0, 78)}`);
112+ liveLine = true;
113+}
114+
115+const plural = (n: number, word: string): string => `${n} ${word}${n === 1 ? '' : 's'}`;
116+
117+function duration(seconds: number): string {
118+ if (!Number.isFinite(seconds)) return '?';
119+ if (seconds < 90) return `${seconds.toFixed(0)}s`;
120+ const m = Math.floor(seconds / 60);
121+ return m < 90 ? `${m}m${String(Math.round(seconds - 60 * m)).padStart(2, '0')}s` : `${(m / 60).toFixed(1)}h`;
122+}
123+
124+/** Parameters in the order the app lists them, not the order they were built. */
125+const paramList = (params: Record<string, number>, choices: DiscreteChoice[]): string =>
126+ choices
127+ .filter((c) => c.key in params)
128+ .map((c) => `${c.key}=${fmtChoice(params[c.key])}`)
129+ .join(' ');
130+
131+/** What a target is, in one line. */
132+function describe(target: AutoTarget): string {
133+ const geomChoices = GEOMETRY_CHOICES[target.geometry] ?? [];
134+ const geom = geomChoices.length
135+ ? `${target.geometry} ${paramList(target.geometryParams, geomChoices)}`
136+ : target.geometry;
137+ return (
138+ `${target.model} ${paramList(target.params, MODEL_CHOICES[target.model])} · ${geom} · ` +
139+ `${plural(target.distance, 'knob')} from the defaults`
140+ );
141+}
142+
143+const doneLine = (run: RunSummary): string =>
144+ `computed in ${duration(run.seconds)}` +
145+ (run.warmFrom !== null ? ` (resumed from cached t = ${fmtChoice(run.warmFrom)})` : '');
146+
147+// ---------------------------------------------------------------- commands
148+async function login(): Promise<void> {
149+ const key = await promptSecret('upload API key: ');
150+ if (!key) throw new Error('nothing entered');
151+ let ok: boolean;
152+ try {
153+ ok = await verifyApiKey(key);
154+ } catch (e) {
155+ say(`could not reach the upload service to check the key (${errMsg(e)}) — saving anyway.`);
156+ ok = true;
157+ }
158+ if (!ok) throw new Error('that key is not allowed to upload — nothing saved');
159+ say(`key saved to ${await saveKey(key)}`);
160+}
161+
162+async function dryRun(opts: Options, targets: AutoTarget[]): Promise<void> {
163+ const shown = targets.slice(0, opts.dryRun);
164+ say(`the first ${plural(shown.length, 'target')} of ${targets.length.toLocaleString()}, ` +
165+ `nearest the defaults first:`);
166+ say();
167+ let cached = 0;
168+ // A handful at a time: a HEAD apiece, and the answers are wanted in order.
169+ const width = String(shown.length).length;
170+ for (let i = 0; i < shown.length; i += 8) {
171+ const batch = shown.slice(i, i + 8);
172+ const present = await Promise.all(
173+ batch.map(async (t) => (await headCached(await lookupFor(specForTarget(t)))) === true),
174+ );
175+ present.forEach((isThere, k) => {
176+ if (isThere) cached++;
177+ say(
178+ ` [${String(i + k + 1).padStart(width)}] ${isThere ? 'cached ' : 'missing'} ` +
179+ describe(batch[k]),
180+ );
181+ });
182+ }
183+ say();
184+ say(`${cached} of ${shown.length} already cached; the walk would compute the other ` +
185+ `${shown.length - cached}.`);
186+}
187+
188+async function fill(opts: Options, targets: AutoTarget[], apiKey: string): Promise<void> {
189+ const runtime = await installWebGpu();
190+ const device = await requestShtDevice().catch((e: unknown) => {
191+ throw new Error(`${errMsg(e)}\n${NO_ADAPTER_HINT}`);
192+ });
193+ const adapter = await describeAdapter(device);
194+ say(`${runtime} · ${adapter}`);
195+ say(`uploads enabled (key ${maskKey(apiKey)})`);
196+ if (isSoftwareAdapter(adapter)) {
197+ say('');
198+ say(`WARNING: ${adapter} is a software rasterizer, not a GPU. Runs here are`);
199+ say(' perhaps a thousand times slower than on hardware — fast enough to look');
200+ say(' like it is working, slow enough to be worth nothing. Check that the');
201+ say(' machine has a GPU and its driver, or stop now.');
202+ }
203+ say('');
204+ say(`${targets.length.toLocaleString()} targets, nearest the defaults first; ` +
205+ `ctrl-C stops after the current run.`);
206+ say('');
207+
208+ const solver = new SolverSession(device, 1, {
209+ onCompiling: (m) => say(` compiling ${m.label}…`),
210+ });
211+
212+ let stopping = false;
213+ process.on('SIGINT', () => {
214+ if (stopping) process.exit(130);
215+ stopping = true;
216+ say('');
217+ say('stopping after this run — ctrl-C again to give up on it.');
218+ });
219+
220+ let index = 0;
221+ let computed = 0;
222+ let uploads = 0;
223+ let slowNoted = false;
224+ let lastLog = 0;
225+ const counts = await fillWalk({
226+ targets,
227+ solver,
228+ adapter,
229+ apiKey: () => apiKey,
230+ beforeTarget: (target) => {
231+ index++;
232+ return specForTarget(target);
233+ },
234+ events: {
235+ onTarget: (target) => say(`[${index}] ${describe(target)}`),
236+ onCached: () => say(' already cached'),
237+ onComputing: (_target, spec) =>
238+ say(` computing to t = ${fmtChoice(spec.tEnd)} ` +
239+ `(${stepsFor(spec).toLocaleString()} steps)`),
240+ onPhase: (phase) => {
241+ if (phase.kind === 'warm-search') say(' looking for a shorter cached run…');
242+ else if (phase.kind === 'seeding') say(' seeding…');
243+ else if (phase.kind === 'uploading') {
244+ say(` ${doneLine(phase.run)} — uploading ${plural(phase.started, 'file')}…`);
245+ }
246+ },
247+ onProgress: (p) => {
248+ const eta = p.rate > 0 ? (p.totalSteps - p.steps) / p.rate : Infinity;
249+ const line =
250+ ` t = ${p.t.toFixed(1)} / ${fmtChoice(p.tEnd)} ` +
251+ `${(100 * p.fraction).toFixed(0)}% ${p.rate.toFixed(0)} steps/s ` +
252+ `eta ${duration(eta)}` +
253+ (p.uploadsStarted ? ` uploaded ${p.uploadsDone}/${p.uploadsStarted}` : '');
254+ live(line);
255+ if (!tty && performance.now() - lastLog > LOG_EVERY_MS) {
256+ lastLog = performance.now();
257+ say(line);
258+ }
259+ // A rate this low means the run is on a software rasterizer, or on a
260+ // GPU so busy it may as well be. Said once, not every chunk.
261+ if (!slowNoted && p.rate > 0 && p.rate < 20 && p.steps > 500) {
262+ slowNoted = true;
263+ say(` NOTE: ${p.rate.toFixed(1)} steps/s is far below what a GPU does ` +
264+ `(${adapter}).`);
265+ }
266+ },
267+ onUploaded: () => uploads++,
268+ onOutcome: (_target, _spec, outcome) => {
269+ if (outcome.kind === 'done') {
270+ computed++;
271+ const times = [...outcome.uploaded].sort((a, b) => a - b).map(fmtChoice).join(', ');
272+ say(` ${doneLine(outcome)} — ` +
273+ (outcome.uploaded.length
274+ ? `uploaded ${plural(outcome.uploaded.length, 'solution')} (t = ${times})`
275+ : 'nothing uploaded'));
276+ for (const err of outcome.uploadErrors) say(` upload failed: ${err}`);
277+ } else if (outcome.kind === 'diverged') {
278+ say(` diverged at t = ${outcome.t.toFixed(2)} — discarded, nothing uploaded`);
279+ } else if (outcome.kind === 'stopped') {
280+ say(` stopped at t = ${outcome.t.toFixed(2)}`);
281+ }
282+ if (computed >= opts.limit) stopping = true;
283+ },
284+ onFailure: (_target, spec, e) => {
285+ const model = mModelByKey(spec.model);
286+ say(` failed: ${formatFailure(e, model?.source ?? '')}`);
287+ },
288+ walkStopped: () => stopping,
289+ stopRequested: () => stopping,
290+ },
291+ });
292+
293+ say('');
294+ say(`stopped — computed ${counts.computed}, skipped ${counts.skipped} already cached` +
295+ (counts.failed ? `, ${counts.failed} failed` : '') +
296+ `; ${plural(uploads, 'file')} uploaded.`);
297+ solver.destroy();
298+ device.destroy();
299+}
300+
301+// ---------------------------------------------------------------- main
302+async function main(): Promise<void> {
303+ const opts = parseArgs(process.argv.slice(2));
304+ if (opts.command === 'help') {
305+ say(HELP);
306+ return;
307+ }
308+ if (opts.command === 'login') {
309+ await login();
310+ return;
311+ }
312+ const targets = autoOrder().filter((t) => !opts.model || t.model === opts.model);
313+ say('turing-surface-cache — the shared cache of Turing patterns on curved surfaces');
314+ if (opts.dryRun) {
315+ await dryRun(opts, targets);
316+ return;
317+ }
318+ const apiKey = await resolveKey(opts.key);
319+ if (!apiKey) {
320+ throw new Error(
321+ 'no upload key. The walk contributes solutions, so it needs one:\n' +
322+ ` ${KEY_ENV}=… npx <this command>\n` +
323+ `or save one for later runs with \`login\` (kept in ${keyPath()}).`,
324+ );
325+ }
326+ await fill(opts, targets, apiKey);
327+}
328+
329+main().catch((e: unknown) => {
330+ say('');
331+ say(errMsg(e));
332+ process.exitCode = 1;
333+});
src/cli/key.tsadded+81−0View file
@@ -0,0 +1,81 @@
1+/**
2+ * Where the command line finds an upload key, and how it saves one.
3+ *
4+ * Three places, in order: the --key option, the environment, and a file the
5+ * `login` subcommand writes. The environment is what the page's copyable
6+ * command uses, since a key on the command line is visible to every user on
7+ * the machine through `ps` while another process's environment is not.
8+ */
9+import { homedir } from 'node:os';
10+import { join } from 'node:path';
11+import { mkdir, readFile, writeFile } from 'node:fs/promises';
12+
13+export const KEY_ENV = 'TURING_SURFACE_CACHE_KEY';
14+
15+const configDir = (): string =>
16+ join(process.env.XDG_CONFIG_HOME || join(homedir(), '.config'), 'turing-surface-cache');
17+
18+export const keyPath = (): string => join(configDir(), 'key');
19+
20+/** The saved key, or '' if there is none. */
21+export async function savedKey(): Promise<string> {
22+ try {
23+ return (await readFile(keyPath(), 'utf8')).trim();
24+ } catch {
25+ return '';
26+ }
27+}
28+
29+/** Save a key for later runs, readable only by this user. */
30+export async function saveKey(key: string): Promise<string> {
31+ await mkdir(configDir(), { recursive: true, mode: 0o700 });
32+ const path = keyPath();
33+ await writeFile(path, `${key}\n`, { mode: 0o600 });
34+ return path;
35+}
36+
37+/** --key, else the environment, else the saved key. */
38+export async function resolveKey(fromOption: string | undefined): Promise<string> {
39+ return (fromOption || process.env[KEY_ENV] || (await savedKey()) || '').trim();
40+}
41+
42+/** Enough of a key to recognize it by, and no more. */
43+export const maskKey = (key: string): string =>
44+ key.length <= 4 ? '·'.repeat(key.length) : `····${key.slice(-4)}`;
45+
46+/**
47+ * Read a secret from the terminal without echoing it. A piped stdin is read as
48+ * a plain line, so `echo $KEY | … login` works too.
49+ */
50+export async function promptSecret(prompt: string): Promise<string> {
51+ const stdin = process.stdin;
52+ process.stdout.write(prompt);
53+ if (!stdin.isTTY) {
54+ const chunks: Buffer[] = [];
55+ for await (const chunk of stdin) chunks.push(chunk as Buffer);
56+ process.stdout.write('\n');
57+ return Buffer.concat(chunks).toString('utf8').split('\n')[0].trim();
58+ }
59+ return new Promise<string>((resolve, reject) => {
60+ let typed = '';
61+ stdin.setRawMode(true);
62+ stdin.resume();
63+ stdin.setEncoding('utf8');
64+ const done = (finish: () => void): void => {
65+ stdin.setRawMode(false);
66+ stdin.pause();
67+ stdin.removeListener('data', onData);
68+ process.stdout.write('\n');
69+ finish();
70+ };
71+ const onData = (chunk: string): void => {
72+ for (const ch of chunk) {
73+ if (ch === '\r' || ch === '\n') return done(() => resolve(typed.trim()));
74+ if (ch === '\u0003') return done(() => reject(new Error('cancelled')));
75+ if (ch === '\u007f' || ch === '\b') typed = typed.slice(0, -1);
76+ else if (ch >= ' ') typed += ch;
77+ }
78+ };
79+ stdin.on('data', onData);
80+ });
81+}
src/cli/webgpu.tsadded+78−0View file
@@ -0,0 +1,78 @@
1+/**
2+ * Desktop WebGPU, via the optional `webgpu` package (prebuilt Google Dawn).
3+ *
4+ * Dawn is installed under the globals the transform code expects
5+ * (navigator.gpu, GPUBufferUsage, …), so everything under src/ runs here
6+ * unchanged — including requestShtDevice(), which makes the same device
7+ * request the browser makes. This file is the whole of what the command line
8+ * has that the page does not.
9+ *
10+ * Adapted from turing-surface's scripts/nodeWebGpu.ts.
11+ */
12+
13+export const errMsg = (e: unknown): string => (e instanceof Error ? e.message : String(e));
14+
15+/** Installs Dawn and returns a human-readable runtime description. */
16+export async function installWebGpu(): Promise<string> {
17+ // The import specifier is indirect so that a build (and a type-check)
18+ // does not require the optional package.
19+ const specifier = 'webgpu';
20+ let mod: {
21+ create: (flags: string[]) => GPU;
22+ globals: Record<string, unknown>;
23+ };
24+ try {
25+ mod = await import(/* @vite-ignore */ specifier);
26+ } catch (e) {
27+ // Distinguish "not installed" from "installed but the prebuilt Dawn binary
28+ // will not load" — the second is what a machine missing a system library
29+ // looks like, and reporting it as the first sends people in circles.
30+ const detail = errMsg(e);
31+ if (/Cannot find (package|module) '?webgpu'?/.test(detail)) {
32+ throw new Error(
33+ 'desktop WebGPU needs the optional `webgpu` package (prebuilt Google Dawn),\n' +
34+ 'which npm may have skipped silently. `npm ls webgpu` says whether it is\n' +
35+ 'there; `npm install webgpu` installs it.',
36+ );
37+ }
38+ const glibc = /GLIBC_([0-9.]+)/.exec(detail);
39+ throw new Error(
40+ `the \`webgpu\` package is installed but did not load:\n ${detail}\n` +
41+ (glibc
42+ ? `Dawn's prebuilt binary wants glibc ${glibc[1]} or newer and this host is older\n` +
43+ '(`ldd --version` says how old). No flag bridges that — use a container with a\n' +
44+ 'newer base image, or a newer host.'
45+ : 'That is usually the prebuilt Dawn binary missing a system library.'),
46+ );
47+ }
48+ Object.assign(globalThis, mod.globals);
49+ // DAWN_FLAGS is ';'-separated because individual Dawn options take
50+ // comma-separated lists, e.g. 'enable-dawn-features=allow_unsafe_apis,...'
51+ const dawnFlags = process.env.DAWN_FLAGS?.split(';').filter(Boolean) ?? [];
52+ Object.defineProperty(globalThis, 'navigator', {
53+ value: { gpu: mod.create(dawnFlags) },
54+ configurable: true,
55+ writable: true,
56+ });
57+ const { version } = await import(/* @vite-ignore */ `${specifier}/package.json`, {
58+ with: { type: 'json' },
59+ }).then(
60+ (m) => m.default as { version: string },
61+ () => ({ version: '?' }),
62+ );
63+ return `node-webgpu ${version} (Google Dawn)`;
64+}
65+
66+/** The hint to print when Dawn loads but finds no adapter. */
67+export const NO_ADAPTER_HINT =
68+ ' Dawn reaches the GPU through Vulkan on Linux and Windows, Metal on macOS,\n' +
69+ " so a headless box may have no adapter at all. DAWN_FLAGS='backend=vulkan'\n" +
70+ ' makes it explain itself.';
71+
72+/**
73+ * Adapters that are not really GPUs. A run on one of these is perhaps a
74+ * thousand times slower than on hardware, which is fast enough to look like it
75+ * is working and slow enough to be worthless — so it is worth saying out loud.
76+ */
77+export const isSoftwareAdapter = (name: string): boolean =>
78+ /swiftshader|llvmpipe|lavapipe|software|microsoft basic|warp/i.test(name);
src/main.tsmodified+10−9View file
@@ -46,7 +46,6 @@ import {
4646 LMAX,
4747 NITER,
4848 LAM3,
49- AUTO_SEED,
5049 AUTO_DT,
5150 defaultChoiceParams,
5251 fmtChoice,
@@ -54,7 +53,7 @@ import {
5453 } from './cache/options.ts';
5554 import { stepsFor, type CacheSpec, APP_NAME, FORMAT_VERSION } from './cache/spec.ts';
5655 import { lookupFor, fetchCached, headCached, type CacheLookup } from './cache/client.ts';
57-import { autoOrder, type AutoTarget } from './cache/autoWalk.ts';
56+import { autoOrder, specForTarget, type AutoTarget } from './cache/autoWalk.ts';
5857 import { decodeCacheFile } from './cache/h5file.ts';
5958 import { SolverSession } from './cache/solver.ts';
6059 import { runSpec, type RunEvents, type RunOutcome, type RunSummary } from './cache/runSpec.ts';
@@ -299,25 +298,27 @@ function resetDefaults(): void {
299298
300299 /** Point every control at one walk target (auto mode drives the same
301300 * selection the user otherwise would, so the URL and the dropdowns always
302- * say what is being computed). */
301+ * say what is being computed). The values come from the target's own spec,
302+ * so currentSpec() reproduces exactly what the walk asked for. */
303303 function setSelection(t: AutoTarget): void {
304- const nextModel = mModelByKey(t.model)!;
304+ const spec = specForTarget(t);
305+ const nextModel = mModelByKey(spec.model)!;
305306 if (nextModel !== model) {
306307 model = nextModel;
307308 elModel.value = model.key;
308309 buildModelParamControls();
309310 }
310- params = { ...t.params };
311- const nextGeom = mGeometryByKey(t.geometry)!;
311+ params = { ...spec.params };
312+ const nextGeom = mGeometryByKey(spec.geometry)!;
312313 if (nextGeom !== geometry) {
313314 geometry = nextGeom;
314315 elGeometry.value = geometry.key;
315316 buildGeomParamControls();
316317 }
317- geomParams = { ...t.geometryParams };
318- seed = AUTO_SEED;
318+ geomParams = { ...spec.geometryParams };
319+ seed = spec.seed;
319320 elSeed.value = String(seed);
320- tEnd = Math.max(...T_END_CHOICE.values);
321+ tEnd = spec.tEnd;
321322 elTend.value = String(tEnd);
322323 syncSelects();
323324 writeUrlState();
src/raw.d.tsmodified+8−0View file
@@ -1,3 +1,11 @@
1+/**
2+ * True in the command line's bundle, false in the page's. Defined by both
3+ * vite configs, so it is a constant the bundler folds away rather than a
4+ * runtime test — the browser build must not carry node's h5wasm, nor the
5+ * command line the browser's (src/cache/h5file.ts).
6+ */
7+declare const __NODE_BUILD__: boolean;
8+
19 /** Vite's `?raw` suffix imports a file's text. Used to load .m model sources. */
210 declare module '*?raw' {
311 const source: string;
vite.cli.config.tsadded+29−0View file
@@ -0,0 +1,29 @@
1+/**
2+ * The command line's bundle: the same source as the page, built for node.
3+ *
4+ * Everything under src/ is bundled in, numbl's compiler included, exactly as
5+ * in the page's build — the published package therefore has no dependency on
6+ * a numbl checkout. What stays external is what cannot be bundled: h5wasm,
7+ * whose node build reads its wasm off disk, and Dawn, which is a native
8+ * addon (and optional, so a machine without it still installs).
9+ */
10+import { defineConfig, mergeConfig } from 'vite';
11+import base from './vite.config.ts';
12+
13+export default mergeConfig(
14+ base,
15+ defineConfig({
16+ define: { __NODE_BUILD__: 'true' },
17+ build: {
18+ ssr: 'src/cli/fill.ts',
19+ outDir: 'dist-cli',
20+ target: 'node22',
21+ emptyOutDir: true,
22+ minify: false,
23+ rollupOptions: {
24+ external: ['h5wasm', 'h5wasm/node', 'webgpu'],
25+ output: { entryFileNames: 'fill.js', banner: '#!/usr/bin/env node' },
26+ },
27+ },
28+ }),
29+);
vite.config.tsmodified+3−0View file
@@ -17,6 +17,9 @@ const numblSrc = realpathSync(resolve(import.meta.dirname, 'node_modules/numbl/s
1717
1818 export default defineConfig({
1919 base: './',
20+ // The page is the browser build; the command line's bundle sets this true
21+ // (vite.cli.config.ts). See src/cache/h5file.ts.
22+ define: { __NODE_BUILD__: 'false' },
2023 resolve: {
2124 alias: { 'numbl-src': numblSrc },
2225 },