Fold the troubleshooting into the page, beside the command - concept-collection/turing-surface-cache

concept-collection / turing-surface-cache

Fold the troubleshooting into the page, beside the command

What goes wrong on somebody else's machine is not guessable from the error it prints, and the page is where they got the command, so it is where the answers belong. A collapsed block under the copy button, shown under the same condition as the command itself, covers the four failures seen so far: a node too old to parse the bundle, a distribution too old for Dawn's prebuilt binary, no GPU for Dawn to reach, and a software rasterizer standing in for one. The second of those is the one worth writing down. Dawn wants glibc 2.34 and a long-lived Linux workstation may be on 2.28, but a container -- the obvious remedy -- does not work: inside one the NVIDIA driver declines to bring up its Vulkan driver, while the identical call on the host succeeds. Borrowing only the userland from a container image and running node through its loader, on the host, does work, and that recipe is now in the page, verified end to end on a Rocky 8 workstation with an RTX 6000 Ada: 444 steps/s, a full 32,000-step target in 74 seconds.

Jeremy Magland <jmagland@flatironinstitute.org> committed 8 days ago commit d71391e85dda parent d3fa654 Browse files

3 changed files+83−0

README.mdmodified+16−0View file

@@ -180,6 +180,22 @@ names its adapter on startup, reports its rate in steps per second, and says

180180 so plainly when either looks wrong; it does not refuse to run, since the

181181 judgment is the operator's.

182182

183+A machine can also be too old for the command in a way that has nothing to do

184+with its GPU. Dawn's prebuilt binary wants glibc 2.34, which a long-lived

185+Linux workstation may well not have — Rocky and RHEL 8 are on 2.28 — and the

186+obvious remedy of running the command in a container turns out not to work:

187+inside one the NVIDIA driver declines to bring up its Vulkan driver

188+(`vk_icdNegotiateLoaderICDInterfaceVersion` returns

189+`VK_ERROR_INITIALIZATION_FAILED`), while the same call on the host succeeds.

190+What does work is to borrow only the userland from a container image and run

191+node through its loader, on the host, leaving the GPU, `/dev` and `/proc`

192+exactly as they were; the host's own `/usr/lib64` stays last on the library

193+path, since the NVIDIA libraries and the Vulkan loader have to match the

194+running kernel module. The page carries that recipe, folded away beside the

195+command it belongs to, along with what the other common failures mean —

196+they are worth writing down where someone will meet them, since none of them

197+is guessable from the error alone.

198+

183199 Progress is a line per target and a rate that updates in place:

184200

185201 ```

index.htmlmodified+65−0View file

@@ -96,6 +96,19 @@

9696 overflow-x: auto; white-space: nowrap;

9797 }

9898 #clicopied { color: var(--ok); }

99+ /* Troubleshooting, folded away: the failures below are undiscoverable

100+ from the error alone, and the page is where people start. */

101+ #clihelp { margin-top: 4px; font-size: 12.5px; }

102+ #clihelp[hidden] { display: none; }

103+ #clihelp summary { cursor: pointer; color: var(--accent); width: fit-content; }

104+ #clihelp p { margin: 8px 0; max-width: 68ch; }

105+ #clihelp code { font-family: ui-monospace, SFMono-Regular, Menlo, monospace; }

106+ #clihelp pre {

107+ font: 12px/1.55 ui-monospace, SFMono-Regular, Menlo, monospace;

108+ background: var(--sphere-bg); border: 1px solid var(--line);

109+ border-radius: 6px; padding: 8px 10px; overflow-x: auto;

110+ color: var(--ink);

111+ }

99112 #err { color: #b35900; white-space: pre-wrap; font-size: 13px; }

100113 </style>

101114 </head>

@@ -166,6 +179,58 @@

166179 title="Copies the command with your key in it. The key is masked here so that it stays out of screenshots.">Copy command (includes your key)</button>

167180 <span id="clicopied"></span>

168181 </div>

182+ <details id="clihelp" hidden>

183+ <summary>if that command does not run on the other machine</summary>

184+ <p><code>Unexpected token {</code>, or another syntax error, means that

185+ machine's node is too old to read the program at all: node 18 or newer

186+ is needed. nodejs.org has current builds, and nvm, fnm and asdf install

187+ one per user without touching what the system depends on; where a

188+ cluster uses environment modules, <code>module load node</code> may be

189+ all it takes.</p>

190+ <p><code>the webgpu package is installed but did not load … GLIBC_2.34

191+ not found</code> means the machine's distribution is older than the

192+ prebuilt Dawn binary — ordinary on a long-lived Linux workstation, since

193+ Rocky and RHEL 8 have glibc 2.28 and Dawn wants 2.34. Running the

194+ command inside a container is the obvious answer and the wrong one:

195+ there the NVIDIA driver declines to bring up its Vulkan driver, while

196+ on the host it comes up fine. What does work is borrowing a newer

197+ userland from a container image and running node through <em>that</em>

198+ loader, still on the host, so the GPU, <code>/dev</code> and

199+ <code>/proc</code> are exactly what they were:</p>

200+ <pre># 1. borrow a userland — podman or docker unpacks it; nothing runs inside it

201+mkdir -p ~/turing-userland

202+podman create --name u docker.io/library/node:22-bookworm-slim true

203+podman export u | tar -x -C ~/turing-userland

204+podman rm u

205+

206+# 2. a node that uses it, running on the host rather than in a container

207+U=~/turing-userland; mkdir -p ~/turing-bin

208+printf '#!/bin/sh\nexec %s --library-path %s %s "$@"\n' \

209+ $U/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 \

210+ $U/usr/lib/x86_64-linux-gnu:$U/lib/x86_64-linux-gnu:/usr/lib64 \

211+ $U/usr/local/bin/node > ~/turing-bin/node

212+chmod +x ~/turing-bin/node

213+

214+# 3. run the command above, with that node found first

215+export PATH=~/turing-bin:$PATH</pre>

216+ <p>The host's own <code>/usr/lib64</code> stays last on that library

217+ path, and is where the NVIDIA libraries and the Vulkan loader come

218+ from: those have to match the running kernel module, so they must be

219+ the machine's own. If podman's storage lives on NFS it will fail to

220+ unpack the image, which

221+ <code>--root /tmp/$USER/store --runroot /tmp/$USER/run</code> on each

222+ podman command works around.</p>

223+ <p><code>WebGPU is not available</code>, or no adapter at all, means

224+ there is no GPU to reach: Dawn goes through Vulkan on Linux and Metal

225+ on macOS, so a machine without a GPU, or with one whose driver is not

226+ installed, has nothing to offer.

227+ <code>DAWN_FLAGS='backend=vulkan'</code> makes it say more about why.</p>

228+ <p>A rate of a few steps per second rather than a few hundred means a

229+ software rasterizer has stood in for the GPU. The command names its

230+ adapter on the first line and says so when it notices, but it does not

231+ refuse to run — a slow contribution is still a contribution, if that is

232+ what you meant.</p>

233+ </details>

169234 </div>

170235 <p id="err"></p>

171236 </main>

src/main.tsmodified+2−0View file

@@ -86,6 +86,7 @@ const elCliBar = $('clibar');

8686 const elCliCmd = $('clicmd');

8787 const elCliCopy = $<HTMLButtonElement>('clicopy');

8888 const elCliCopied = $('clicopied');

89+const elCliHelp = $('clihelp');

8990 const elErr = $('err');

9091

9192 /**

@@ -1041,6 +1042,7 @@ function updateUploadNote(): void {

10411042 // and so is the command that does the same thing elsewhere.

10421043 elAutoBar.hidden = !hasKey;

10431044 elCliBar.hidden = !hasKey;

1045+ elCliHelp.hidden = !hasKey;

10441046 elCliCmd.textContent = fillCommand('…');

10451047 elCliCopied.textContent = '';

10461048 if (!hasKey && autoRunning) autoRunning = false;