Demonstrate lazy reading of remote HDF5/NWB files (direct h5wasm + LINDI)
32 changed files+7287−0
.github/workflows/deploy.ymladded+54−0View file
@@ -0,0 +1,54 @@
1+name: Deploy to GitHub Pages
2+
3+on:
4+ push:
5+ branches:
6+ - main
7+ workflow_dispatch:
8+
9+permissions:
10+ contents: read
11+ pages: write
12+ id-token: write
13+
14+concurrency:
15+ group: "pages"
16+ cancel-in-progress: false
17+
18+jobs:
19+ build:
20+ runs-on: ubuntu-latest
21+ steps:
22+ - name: Checkout
23+ uses: actions/checkout@v4
24+
25+ - name: Setup Node
26+ uses: actions/setup-node@v4
27+ with:
28+ node-version: "20"
29+ cache: "npm"
30+
31+ - name: Install dependencies
32+ run: npm ci
33+
34+ - name: Build
35+ run: npm run build
36+
37+ - name: Setup Pages
38+ uses: actions/configure-pages@v4
39+
40+ - name: Upload artifact
41+ uses: actions/upload-pages-artifact@v3
42+ with:
43+ path: "./dist"
44+
45+ deploy:
46+ environment:
47+ name: github-pages
48+ url: ${{ steps.deployment.outputs.page_url }}
49+ runs-on: ubuntu-latest
50+ needs: build
51+ steps:
52+ - name: Deploy to GitHub Pages
53+ id: deployment
54+ uses: actions/deploy-pages@v4
.gitignoreadded+5−0View file
@@ -0,0 +1,5 @@
1+node_modules
2+dist
3+*.local
4+.DS_Store
5+.vite
README.mdadded+125−0View file
@@ -0,0 +1,125 @@
1+# Lazy reading of remote HDF5 / NWB files
2+
3+This is a small demonstration of how [neurosift](https://neurosift.app/) browses
4+large [NWB](https://www.nwb.org/) files directly in the browser, pulling data out
5+of HDF5 files that live on remote storage without downloading them in full and
6+without any backend server.
7+
8+The [live demo](?#demo) opens one NWB file from
9+[DANDI dandiset 000986](https://dandiarchive.org/dandiset/000986) (recordings from
10+mouse auditory cortex) straight from DANDI's S3 bucket. It reads the session
11+metadata, walks the top of the file's group structure, and pulls a short window
12+out of a multi-million-sample timeseries, all on demand.
13+
14+## Why this is possible
15+
16+An HDF5 file is not a blob you have to read start to finish. It is a small amount
17+of structural metadata (a superblock, some B-trees, object headers) together with
18+the array data laid out in independently addressable chunks. If you know which
19+byte ranges hold the thing you want, you can read just those bytes and ignore the
20+rest.
21+
22+That maps neatly onto an HTTP feature that has been around forever: the `Range`
23+request. DANDI's S3 objects honor range requests, so a browser can treat a remote
24+multi-gigabyte NWB file as if it were local, fetching a few kilobytes here and
25+there as it goes. Opening the file, expanding a group, or slicing a dataset each
26+turn into a handful of small requests rather than a download.
27+
28+There are two ways neurosift does the reading, and the demo runs both next to each
29+other so you can compare them.
30+
31+## Reading the HDF5 directly
32+
33+The first approach reads the actual HDF5 file. The HDF5 C library is compiled to
34+WebAssembly ([h5wasm](https://github.com/usnistgov/h5wasm)) and run inside a web
35+worker. The worker hands h5wasm a file that is backed by the network rather than
36+by disk, using emscripten's lazy filesystem:
37+
38+```js
39+// remote-h5-worker
40+FS.createLazyFile('/', fname, url, true, false, headers, chunkSize);
41+const file = new h5wasm.File(fname);
42+```
43+
44+Whenever h5wasm tries to read some offset in that file, the lazy filesystem
45+fetches the chunk that contains it over HTTP and caches it. So the parsing is done
46+by the real HDF5 library (the same code you would run locally), but the bytes
47+trickle in from S3 as the library walks the structure. The worker exposes three
48+calls, `getGroup`, `getDataset`, and `getDatasetData(path, { slice })`, and the
49+main thread talks to it through a thin wrapper (`RemoteH5File`) that caches
50+results.
51+
52+The catch is latency. Walking HDF5's B-trees can take a lot of small round trips
53+before you have the metadata you need, and over a network that adds up.
54+
55+## Reading through a LINDI index
56+
57+The second approach side-steps that latency.
58+[LINDI](https://github.com/neurodatawithoutborders/lindi) precomputes the answer
59+to "where is everything?" once, on the server, and stores it as a single JSON
60+file. The format follows the [kerchunk](https://github.com/fsspec/kerchunk)
61+convention and is, in fact, a valid Zarr store. It is a dictionary of `refs` whose
62+keys are Zarr paths and whose values are one of:
63+
64+```jsonc
65+{
66+ "refs": {
67+ "units/.zgroup": "{\"zarr_format\":2}", // small data stored inline
68+ "units/spike_times/.zarray": { /* shape, dtype, compressor, ... */ },
69+ "units/spike_times/0": ["<original-hdf5-url>", 12345, 678] // [url, offset, length]
70+ }
71+}
72+```
73+
74+Group structure, attributes, and small datasets are inlined, while large array
75+chunks are left as `[url, offset, length]` references that point straight back
76+into the original HDF5 file on S3. LINDI also defines a few extra Zarr annotations
77+(`_SCALAR`, `_REFERENCE`, `_COMPOUND_DTYPE`, `_EXTERNAL_ARRAY_LINK`) so it can
78+faithfully represent HDF5 features that plain Zarr has no notion of, such as scalar
79+datasets, object references, and compound types.
80+
81+The payoff is that the entire structure of the file arrives in one request. After
82+that, reading actual data still uses range requests against the original HDF5,
83+with the chunks decoded client-side (blosc, zlib, and friends). neurosift keeps
84+pre-generated indexes for published dandisets at `lindi.neurosift.org` and uses
85+one when it is available, falling back to direct h5wasm reading when it is not.
86+
87+## Putting it together
88+
89+```
90+NwbPage ── hdf5Interface ──► RemoteH5File ─RPC→ worker ─► h5wasm ─Range→ S3
91+ └► RemoteH5FileLindi ──► JSON index, then Range→ S3
92+```
93+
94+A DANDI asset URL is first followed to its underlying S3 object. neurosift then
95+prefers the LINDI index if one exists and otherwise reads the raw HDF5 through the
96+h5wasm worker. Either way, the viewers downstream only ask for the slices they
97+actually draw, which is what keeps even very large files responsive.
98+
99+The reading library under `src/remote-h5-file` is taken unchanged from neurosift,
100+and the worker is loaded from `tempory.net/js/RemoteH5Worker.js`, built from
101+[magland/remote-h5-worker](https://github.com/magland/remote-h5-worker).
102+
103+## About the demo
104+
105+The two panels read the same file using the two strategies above. Each one reports
106+how long the open took, lists the root group, shows a handful of session and
107+subject fields, and then loads the first 30,000 of roughly 7.5 million samples of
108+the pupil-diameter and running-speed traces (well under one percent of the data)
109+before plotting them. It finishes by reading the trials table and plotting stimulus
110+frequency across the session. If you open the network tab while it runs, you can
111+watch the partial range requests come back.
112+
113+## Running locally
114+
115+```bash
116+npm install
117+npm run dev
118+```
119+
120+## Credits
121+
122+Built on [neurosift](https://github.com/flatironinstitute/neurosift),
123+[h5wasm](https://github.com/usnistgov/h5wasm), and
124+[LINDI](https://github.com/neurodatawithoutborders/lindi). Example data is
125+[DANDI:000986](https://dandiarchive.org/dandiset/000986).
index.htmladded+12−0View file
@@ -0,0 +1,12 @@
1+<!doctype html>
2+<html lang="en">
3+ <head>
4+ <meta charset="UTF-8" />
5+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
6+ <title>Lazy reading of remote HDF5 / NWB</title>
7+ </head>
8+ <body>
9+ <div id="root"></div>
10+ <script type="module" src="/src/main.tsx"></script>
11+ </body>
12+</html>
package-lock.jsonadded+3357−0View file
This diff is 3,362 lines long and is not shown.
package.jsonadded+27−0View file
@@ -0,0 +1,27 @@
1+{
2+ "name": "remote-hdf5-lazy-read",
3+ "private": true,
4+ "version": "0.0.0",
5+ "type": "module",
6+ "scripts": {
7+ "dev": "vite",
8+ "build": "tsc -b && vite build",
9+ "preview": "vite preview"
10+ },
11+ "dependencies": {
12+ "numcodecs": "^0.3.2",
13+ "pako": "^2.1.0",
14+ "react": "^19.2.0",
15+ "react-dom": "^19.2.0",
16+ "react-markdown": "^10.1.0",
17+ "remark-gfm": "^4.0.1"
18+ },
19+ "devDependencies": {
20+ "@types/pako": "^2.0.4",
21+ "@types/react": "^19.2.5",
22+ "@types/react-dom": "^19.2.3",
23+ "@vitejs/plugin-react": "^5.1.1",
24+ "typescript": "~5.9.3",
25+ "vite": "^7.2.4"
26+ }
27+}
src/App.tsxadded+71−0View file
@@ -0,0 +1,71 @@
1+import { useState } from "react";
2+import { MarkdownPage } from "./components/MarkdownPage";
3+import { Demo } from "./components/Demo";
4+
5+type Tab = "about" | "demo";
6+
7+export const App = () => {
8+ const [tab, setTab] = useState<Tab>("about");
9+
10+ return (
11+ <div
12+ style={{
13+ maxWidth: 980,
14+ margin: "0 auto",
15+ padding: "0 20px 60px",
16+ fontFamily:
17+ "-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Helvetica, Arial, sans-serif",
18+ color: "#1f2328",
19+ }}
20+ >
21+ <nav
22+ style={{
23+ position: "sticky",
24+ top: 0,
25+ background: "#fff",
26+ borderBottom: "1px solid #e2e2e2",
27+ padding: "12px 0",
28+ display: "flex",
29+ gap: 8,
30+ zIndex: 10,
31+ }}
32+ >
33+ <TabButton active={tab === "about"} onClick={() => setTab("about")}>
34+ About
35+ </TabButton>
36+ <TabButton active={tab === "demo"} onClick={() => setTab("demo")}>
37+ Live demo
38+ </TabButton>
39+ </nav>
40+ <div style={{ paddingTop: 20 }}>
41+ {tab === "about" ? <MarkdownPage /> : <Demo />}
42+ </div>
43+ </div>
44+ );
45+};
46+
47+const TabButton = ({
48+ active,
49+ onClick,
50+ children,
51+}: {
52+ active: boolean;
53+ onClick: () => void;
54+ children: React.ReactNode;
55+}) => (
56+ <button
57+ onClick={onClick}
58+ style={{
59+ border: "none",
60+ background: active ? "#1f6feb" : "transparent",
61+ color: active ? "#fff" : "#1f6feb",
62+ padding: "6px 14px",
63+ borderRadius: 6,
64+ cursor: "pointer",
65+ fontSize: 14,
66+ fontWeight: 600,
67+ }}
68+ >
69+ {children}
70+ </button>
71+);
src/components/BackendPanel.tsxadded+195−0View file
@@ -0,0 +1,195 @@
1+import { useBackend } from "../nwb/useBackend";
2+import type { BackendMode } from "../nwb/useBackend";
3+import { useNwbData, WINDOW_SAMPLES } from "../nwb/useNwbData";
4+import { LinePlot } from "./LinePlot";
5+
6+type Props = {
7+ title: string;
8+ description: string;
9+ mode: BackendMode;
10+ downloadUrl: string;
11+ dandisetId: string;
12+};
13+
14+// Decimate a long array down to ~maxPoints for light SVG rendering.
15+const decimate = <T extends { length: number; [i: number]: number }>(
16+ arr: T,
17+ maxPoints = 1500,
18+): number[] => {
19+ const step = Math.max(1, Math.floor(arr.length / maxPoints));
20+ const out: number[] = [];
21+ for (let i = 0; i < arr.length; i += step) out.push(arr[i]);
22+ return out;
23+};
24+
25+export const BackendPanel = ({
26+ title,
27+ description,
28+ mode,
29+ downloadUrl,
30+ dandisetId,
31+}: Props) => {
32+ const backend = useBackend(downloadUrl, dandisetId, mode);
33+ const data = useNwbData(backend.file);
34+
35+ return (
36+ <div
37+ style={{
38+ flex: "1 1 420px",
39+ minWidth: 380,
40+ border: "1px solid #d0d0d0",
41+ borderRadius: 8,
42+ padding: 16,
43+ background: "#fafafa",
44+ }}
45+ >
46+ <h3 style={{ marginTop: 0 }}>{title}</h3>
47+ <p style={{ color: "#555", fontSize: 13, marginTop: 0 }}>{description}</p>
48+
49+ <div style={{ fontSize: 13, marginBottom: 8 }}>
50+ <strong>Status:</strong>{" "}
51+ {backend.status === "opening" && "opening remote file…"}
52+ {backend.status === "ready" && (
53+ <span style={{ color: "#1a7f37" }}>
54+ opened in {backend.openMs?.toFixed(0)} ms
55+ </span>
56+ )}
57+ {backend.status === "error" && (
58+ <span style={{ color: "#b00020" }}>error</span>
59+ )}
60+ {backend.status === "idle" && "idle"}
61+ </div>
62+
63+ {backend.resolvedUrl && (
64+ <div
65+ style={{
66+ fontSize: 11,
67+ color: "#777",
68+ wordBreak: "break-all",
69+ marginBottom: 8,
70+ }}
71+ >
72+ <strong>Resolved:</strong> {backend.resolvedUrl}
73+ </div>
74+ )}
75+
76+ {backend.error && (
77+ <div style={{ color: "#b00020", fontSize: 13, marginBottom: 8 }}>
78+ {backend.error}
79+ </div>
80+ )}
81+
82+ {backend.status === "ready" && (
83+ <>
84+ <div style={{ fontSize: 13, marginBottom: 8 }}>
85+ <strong>Reading:</strong>{" "}
86+ {data.done ? (
87+ data.error ? (
88+ <span style={{ color: "#b00020" }}>{data.error}</span>
89+ ) : (
90+ <span style={{ color: "#1a7f37" }}>complete</span>
91+ )
92+ ) : (
93+ <span>{data.step}…</span>
94+ )}
95+ </div>
96+
97+ {data.rootGroup && (
98+ <details style={{ marginBottom: 10 }}>
99+ <summary style={{ cursor: "pointer", fontSize: 13 }}>
100+ Root group: {data.rootGroup.subgroups.length} groups,{" "}
101+ {data.rootGroup.datasets.length} datasets
102+ </summary>
103+ <div style={{ fontFamily: "monospace", fontSize: 12, marginTop: 6 }}>
104+ {data.rootGroup.subgroups.map((g) => (
105+ <div key={g.path}>📁 {g.name}/</div>
106+ ))}
107+ {data.rootGroup.datasets.map((d) => (
108+ <div key={d.path}>📄 {d.name}</div>
109+ ))}
110+ </div>
111+ </details>
112+ )}
113+
114+ {data.meta && (
115+ <table style={{ fontSize: 12, marginBottom: 12, lineHeight: 1.5 }}>
116+ <tbody>
117+ {Object.entries(data.meta).map(([k, v]) => (
118+ <tr key={k}>
119+ <td
120+ style={{
121+ color: "#555",
122+ paddingRight: 10,
123+ verticalAlign: "top",
124+ whiteSpace: "nowrap",
125+ }}
126+ >
127+ {k}
128+ </td>
129+ <td>{v || <span style={{ color: "#aaa" }}>—</span>}</td>
130+ </tr>
131+ ))}
132+ </tbody>
133+ </table>
134+ )}
135+
136+ {data.timeseries && (
137+ <div style={{ marginBottom: 12 }}>
138+ <div style={{ fontSize: 12, color: "#555", marginBottom: 4 }}>
139+ Loaded first {data.timeseries.t.length.toLocaleString()} of{" "}
140+ {data.timeseries.totalSamples.toLocaleString()} samples (
141+ {(
142+ (data.timeseries.t.length / (data.timeseries.totalSamples || 1)) *
143+ 100
144+ ).toFixed(2)}
145+ % of the trace — only those bytes were fetched).
146+ </div>
147+ <LinePlot
148+ xLabel="time (s)"
149+ yLabel="value"
150+ series={[
151+ {
152+ x: decimate(data.timeseries.t),
153+ y: decimate(data.timeseries.pupil),
154+ color: "#1f77b4",
155+ label: "pupil diameter",
156+ },
157+ {
158+ x: decimate(data.timeseries.t),
159+ y: decimate(data.timeseries.running),
160+ color: "#d62728",
161+ label: "running speed",
162+ },
163+ ]}
164+ />
165+ </div>
166+ )}
167+
168+ {data.trials && (
169+ <div>
170+ <div style={{ fontSize: 12, color: "#555", marginBottom: 4 }}>
171+ Trials table: {data.trials.stimFrequency.length.toLocaleString()}{" "}
172+ trials — stimulus frequency over the session.
173+ </div>
174+ <LinePlot
175+ xLabel="trial start time (s)"
176+ yLabel="stim frequency (Hz)"
177+ series={[
178+ {
179+ x: decimate(data.trials.startTime),
180+ y: decimate(data.trials.stimFrequency),
181+ color: "#2ca02c",
182+ label: "stim frequency",
183+ },
184+ ]}
185+ />
186+ </div>
187+ )}
188+ </>
189+ )}
190+ <div style={{ fontSize: 11, color: "#999", marginTop: 10 }}>
191+ Window size: {WINDOW_SAMPLES.toLocaleString()} samples
192+ </div>
193+ </div>
194+ );
195+};
src/components/Demo.tsxadded+60−0View file
@@ -0,0 +1,60 @@
1+import { BackendPanel } from "./BackendPanel";
2+
3+// One of the NWB files in DANDI dandiset 000986 (mouse auditory cortex).
4+// This is the same asset shown at:
5+// https://neurosift.app/nwb?url=https%3A%2F%2Fapi.dandiarchive.org%2Fapi%2Fassets%2Faacd1c8a-73f7-469e-bf08-0afd5c1052f9%2Fdownload%2F&dandisetId=000986
6+const DOWNLOAD_URL =
7+ "https://api.dandiarchive.org/api/assets/aacd1c8a-73f7-469e-bf08-0afd5c1052f9/download/";
8+const DANDISET_ID = "000986";
9+
10+export const Demo = () => {
11+ return (
12+ <div>
13+ <h2>Live demo</h2>
14+ <p style={{ fontSize: 14, color: "#444", maxWidth: 800 }}>
15+ Both panels below read the <em>same</em> remote NWB file lazily, entirely
16+ in your browser — nothing is downloaded in full and there is no server.
17+ The left panel reads the raw HDF5 over HTTP range requests (h5wasm); the
18+ right panel reads through a precomputed LINDI JSON index. Open your
19+ browser's network tab to watch the partial range requests.
20+ </p>
21+ <div
22+ style={{
23+ fontSize: 12,
24+ fontFamily: "monospace",
25+ background: "#f0f0f0",
26+ padding: "8px 10px",
27+ borderRadius: 6,
28+ marginBottom: 16,
29+ wordBreak: "break-all",
30+ }}
31+ >
32+ dandiset {DANDISET_ID} · {DOWNLOAD_URL}
33+ <br />
34+ <a
35+ href={`https://neurosift.app/nwb?url=${encodeURIComponent(DOWNLOAD_URL)}&dandisetId=${DANDISET_ID}`}
36+ target="_blank"
37+ rel="noopener noreferrer"
38+ >
39+ open this file in neurosift ↗
40+ </a>
41+ </div>
42+ <div style={{ display: "flex", flexWrap: "wrap", gap: 16 }}>
43+ <BackendPanel
44+ title="Direct HDF5 (h5wasm + range requests)"
45+ description="Resolves the DANDI redirect, then reads the raw .nwb HDF5 with h5wasm in a web worker. Each group/dataset/slice triggers HTTP Range requests for just those bytes."
46+ mode="direct"
47+ downloadUrl={DOWNLOAD_URL}
48+ dandisetId={DANDISET_ID}
49+ />
50+ <BackendPanel
51+ title="LINDI (JSON reference index)"
52+ description="Loads a precomputed JSON index that maps every group/dataset/attribute to byte ranges. Metadata is read from the index; only chunk data is range-requested from the original HDF5."
53+ mode="lindi"
54+ downloadUrl={DOWNLOAD_URL}
55+ dandisetId={DANDISET_ID}
56+ />
57+ </div>
58+ </div>
59+ );
60+};
src/components/LinePlot.tsxadded+156−0View file
@@ -0,0 +1,156 @@
1+type Series = {
2+ x: number[] | Float64Array | Float32Array;
3+ y: number[] | Float64Array | Float32Array;
4+ color: string;
5+ label: string;
6+};
7+
8+type Props = {
9+ series: Series[];
10+ width?: number;
11+ height?: number;
12+ xLabel?: string;
13+ yLabel?: string;
14+};
15+
16+// Minimal dependency-free line plot rendered as SVG.
17+export const LinePlot = ({
18+ series,
19+ width = 560,
20+ height = 220,
21+ xLabel,
22+ yLabel,
23+}: Props) => {
24+ const padL = 48;
25+ const padR = 12;
26+ const padT = 10;
27+ const padB = 34;
28+ const innerW = width - padL - padR;
29+ const innerH = height - padT - padB;
30+
31+ let xMin = Infinity;
32+ let xMax = -Infinity;
33+ let yMin = Infinity;
34+ let yMax = -Infinity;
35+ for (const s of series) {
36+ for (let i = 0; i < s.x.length; i++) {
37+ const xv = s.x[i];
38+ const yv = s.y[i];
39+ if (xv < xMin) xMin = xv;
40+ if (xv > xMax) xMax = xv;
41+ if (yv < yMin) yMin = yv;
42+ if (yv > yMax) yMax = yv;
43+ }
44+ }
45+ if (!isFinite(xMin)) {
46+ xMin = 0;
47+ xMax = 1;
48+ yMin = 0;
49+ yMax = 1;
50+ }
51+ if (xMax === xMin) xMax = xMin + 1;
52+ if (yMax === yMin) yMax = yMin + 1;
53+
54+ const sx = (x: number) => padL + ((x - xMin) / (xMax - xMin)) * innerW;
55+ const sy = (y: number) => padT + innerH - ((y - yMin) / (yMax - yMin)) * innerH;
56+
57+ const pathFor = (s: Series) => {
58+ let d = "";
59+ for (let i = 0; i < s.x.length; i++) {
60+ d += `${i === 0 ? "M" : "L"}${sx(s.x[i]).toFixed(1)},${sy(s.y[i]).toFixed(1)} `;
61+ }
62+ return d;
63+ };
64+
65+ const fmt = (v: number) =>
66+ Math.abs(v) >= 1000 || (v !== 0 && Math.abs(v) < 0.01)
67+ ? v.toExponential(1)
68+ : v.toFixed(2);
69+
70+ return (
71+ <svg
72+ width={width}
73+ height={height}
74+ style={{ background: "#fff", border: "1px solid #e2e2e2" }}
75+ >
76+ {/* axes */}
77+ <line x1={padL} y1={padT} x2={padL} y2={padT + innerH} stroke="#999" />
78+ <line
79+ x1={padL}
80+ y1={padT + innerH}
81+ x2={padL + innerW}
82+ y2={padT + innerH}
83+ stroke="#999"
84+ />
85+ {/* y ticks */}
86+ <text x={padL - 6} y={padT + 4} textAnchor="end" fontSize={10} fill="#555">
87+ {fmt(yMax)}
88+ </text>
89+ <text
90+ x={padL - 6}
91+ y={padT + innerH}
92+ textAnchor="end"
93+ fontSize={10}
94+ fill="#555"
95+ >
96+ {fmt(yMin)}
97+ </text>
98+ {/* x ticks */}
99+ <text
100+ x={padL}
101+ y={padT + innerH + 14}
102+ textAnchor="start"
103+ fontSize={10}
104+ fill="#555"
105+ >
106+ {fmt(xMin)}
107+ </text>
108+ <text
109+ x={padL + innerW}
110+ y={padT + innerH + 14}
111+ textAnchor="end"
112+ fontSize={10}
113+ fill="#555"
114+ >
115+ {fmt(xMax)}
116+ </text>
117+ {/* axis labels */}
118+ {xLabel && (
119+ <text
120+ x={padL + innerW / 2}
121+ y={height - 4}
122+ textAnchor="middle"
123+ fontSize={11}
124+ fill="#333"
125+ >
126+ {xLabel}
127+ </text>
128+ )}
129+ {yLabel && (
130+ <text
131+ x={12}
132+ y={padT + innerH / 2}
133+ textAnchor="middle"
134+ fontSize={11}
135+ fill="#333"
136+ transform={`rotate(-90 12 ${padT + innerH / 2})`}
137+ >
138+ {yLabel}
139+ </text>
140+ )}
141+ {/* data */}
142+ {series.map((s, i) => (
143+ <path key={i} d={pathFor(s)} fill="none" stroke={s.color} strokeWidth={1} />
144+ ))}
145+ {/* legend */}
146+ {series.map((s, i) => (
147+ <g key={`l${i}`}>
148+ <rect x={padL + 8} y={padT + 6 + i * 14} width={10} height={3} fill={s.color} />
149+ <text x={padL + 22} y={padT + 10 + i * 14} fontSize={10} fill="#333">
150+ {s.label}
151+ </text>
152+ </g>
153+ ))}
154+ </svg>
155+ );
156+};
src/components/MarkdownPage.tsxadded+11−0View file
@@ -0,0 +1,11 @@
1+import Markdown from "react-markdown";
2+import remarkGfm from "remark-gfm";
3+import readmeText from "../../README.md?raw";
4+
5+export const MarkdownPage = () => {
6+ return (
7+ <div className="markdown-body">
8+ <Markdown remarkPlugins={[remarkGfm]}>{readmeText}</Markdown>
9+ </div>
10+ );
11+};
src/index.cssadded+60−0View file
@@ -0,0 +1,60 @@
1+body {
2+ margin: 0;
3+ background: #fff;
4+ line-height: 1.5;
5+}
6+
7+.markdown-body {
8+ font-size: 15px;
9+}
10+
11+.markdown-body h1 {
12+ border-bottom: 1px solid #e2e2e2;
13+ padding-bottom: 8px;
14+}
15+
16+.markdown-body h2 {
17+ margin-top: 28px;
18+ border-bottom: 1px solid #eee;
19+ padding-bottom: 4px;
20+}
21+
22+.markdown-body code {
23+ background: #f3f3f3;
24+ padding: 1px 5px;
25+ border-radius: 4px;
26+ font-size: 90%;
27+}
28+
29+.markdown-body pre {
30+ background: #f6f8fa;
31+ padding: 12px;
32+ border-radius: 6px;
33+ overflow-x: auto;
34+}
35+
36+.markdown-body pre code {
37+ background: none;
38+ padding: 0;
39+}
40+
41+.markdown-body a {
42+ color: #1f6feb;
43+}
44+
45+.markdown-body table {
46+ border-collapse: collapse;
47+}
48+
49+.markdown-body th,
50+.markdown-body td {
51+ border: 1px solid #d0d7de;
52+ padding: 6px 12px;
53+}
54+
55+.markdown-body blockquote {
56+ color: #57606a;
57+ border-left: 3px solid #d0d7de;
58+ margin: 0;
59+ padding: 0 14px;
60+}
src/main.tsxadded+10−0View file
@@ -0,0 +1,10 @@
1+import { StrictMode } from "react";
2+import { createRoot } from "react-dom/client";
3+import { App } from "./App";
4+import "./index.css";
5+
6+createRoot(document.getElementById("root")!).render(
7+ <StrictMode>
8+ <App />
9+ </StrictMode>,
10+);
src/nwb/dandi.tsadded+37−0View file
@@ -0,0 +1,37 @@
1+// Helpers for resolving DANDI asset URLs and locating LINDI indexes.
2+// This mirrors what neurosift does in src/pages/NwbPage/hdf5Interface.ts.
3+
4+export const extractAssetId = (downloadUrl: string): string | undefined => {
5+ // e.g. https://api.dandiarchive.org/api/assets/<assetId>/download/
6+ const m = downloadUrl.match(/\/assets\/([^/]+)\/download/);
7+ return m ? m[1] : undefined;
8+};
9+
10+// DANDI download URLs are 302-redirected to a (signed) S3 object URL.
11+// We follow the redirect with a GET and then abort, because a HEAD request is
12+// rejected by CORS on the DANDI S3 bucket. We only want `response.url`.
13+export const resolveDandiDownloadUrl = async (url: string): Promise<string> => {
14+ const controller = new AbortController();
15+ try {
16+ const response = await fetch(url, { signal: controller.signal });
17+ const resolved = response.url || url;
18+ controller.abort();
19+ return resolved;
20+ } catch {
21+ controller.abort();
22+ return url;
23+ }
24+};
25+
26+// neurosift hosts precomputed LINDI indexes for published dandisets here.
27+export const lindiUrlForAsset = (dandisetId: string, assetId: string): string =>
28+ `https://lindi.neurosift.org/dandi/dandisets/${dandisetId}/assets/${assetId}/nwb.lindi.json`;
29+
30+export const lindiIndexExists = async (lindiUrl: string): Promise<boolean> => {
31+ try {
32+ const resp = await fetch(lindiUrl, { method: "HEAD" });
33+ return resp.ok;
34+ } catch {
35+ return false;
36+ }
37+};
src/nwb/useBackend.tsadded+80−0View file
@@ -0,0 +1,80 @@
1+import { useEffect, useState } from "react";
2+import { RemoteH5File, RemoteH5FileLindi } from "../remote-h5-file";
3+import type { RemoteH5FileX } from "../remote-h5-file";
4+import {
5+ extractAssetId,
6+ lindiIndexExists,
7+ lindiUrlForAsset,
8+ resolveDandiDownloadUrl,
9+} from "./dandi";
10+
11+export type BackendMode = "direct" | "lindi";
12+
13+export type BackendState = {
14+ file?: RemoteH5FileX;
15+ resolvedUrl?: string;
16+ openMs?: number;
17+ status: "idle" | "opening" | "ready" | "error";
18+ error?: string;
19+};
20+
21+// Opens the remote NWB file using one of the two strategies:
22+// - "direct": resolve the DANDI redirect, then read the raw HDF5 lazily via
23+// the h5wasm worker (HTTP range requests).
24+// - "lindi": locate the precomputed LINDI JSON index and read metadata from
25+// it, range-reading only chunk data from the original HDF5.
26+export const useBackend = (
27+ downloadUrl: string,
28+ dandisetId: string,
29+ mode: BackendMode,
30+): BackendState => {
31+ const [state, setState] = useState<BackendState>({ status: "idle" });
32+
33+ useEffect(() => {
34+ let canceled = false;
35+ const open = async () => {
36+ setState({ status: "opening" });
37+ const t0 = performance.now();
38+ try {
39+ let file: RemoteH5FileX;
40+ let resolvedUrl: string;
41+ if (mode === "direct") {
42+ resolvedUrl = await resolveDandiDownloadUrl(downloadUrl);
43+ file = new RemoteH5File(resolvedUrl, {});
44+ } else {
45+ const assetId = extractAssetId(downloadUrl);
46+ if (!assetId) throw new Error("Could not extract asset id from URL");
47+ resolvedUrl = lindiUrlForAsset(dandisetId, assetId);
48+ if (!(await lindiIndexExists(resolvedUrl))) {
49+ throw new Error(
50+ "No LINDI index is available for this asset. neurosift would " +
51+ "fall back to direct HDF5 reading in this case.",
52+ );
53+ }
54+ file = await RemoteH5FileLindi.create(resolvedUrl);
55+ }
56+ // Touch the root group so "ready" means metadata is actually reachable.
57+ await file.getGroup("/");
58+ if (canceled) return;
59+ setState({
60+ file,
61+ resolvedUrl,
62+ openMs: performance.now() - t0,
63+ status: "ready",
64+ });
65+ } catch (err: unknown) {
66+ if (canceled) return;
67+ setState({
68+ status: "error",
69+ error: err instanceof Error ? err.message : String(err),
70+ });
71+ }
72+ };
73+ open();
74+ return () => {
75+ canceled = true;
76+ };
77+ }, [downloadUrl, dandisetId, mode]);
78+
79+ return state;
80+};
src/nwb/useNwbData.tsadded+126−0View file
@@ -0,0 +1,126 @@
1+import { useEffect, useState } from "react";
2+import type { RemoteH5FileX, RemoteH5Group } from "../remote-h5-file";
3+
4+// Number of samples we lazily pull from the multi-million-sample timeseries.
5+export const WINDOW_SAMPLES = 30000;
6+
7+export type NwbData = {
8+ rootGroup?: RemoteH5Group;
9+ meta?: Record<string, string>;
10+ timeseries?: {
11+ t: Float64Array;
12+ pupil: Float64Array;
13+ running: Float64Array;
14+ totalSamples: number;
15+ };
16+ trials?: { startTime: Float64Array; stimFrequency: Float64Array };
17+ step: string;
18+ error?: string;
19+ done: boolean;
20+};
21+
22+const PUPIL = "/processing/behavior/PupilTracking/pupil_diameter";
23+const RUNNING = "/processing/behavior/running_speed";
24+const TRIALS = "/intervals/trials";
25+
26+const toF64 = (x: unknown): Float64Array => {
27+ if (x instanceof Float64Array) return x;
28+ if (ArrayBuffer.isView(x)) return Float64Array.from(x as unknown as number[]);
29+ if (Array.isArray(x)) return Float64Array.from(x as number[]);
30+ return new Float64Array(0);
31+};
32+
33+const readScalar = async (
34+ file: RemoteH5FileX,
35+ path: string,
36+): Promise<string> => {
37+ try {
38+ const v = await file.getDatasetData(path, {});
39+ if (v === undefined || v === null) return "";
40+ if (typeof v === "string") return v;
41+ if (ArrayBuffer.isView(v) || Array.isArray(v)) {
42+ return Array.from(v as unknown as number[]).join(", ");
43+ }
44+ return String(v);
45+ } catch {
46+ return "";
47+ }
48+};
49+
50+export const useNwbData = (file: RemoteH5FileX | undefined): NwbData => {
51+ const [data, setData] = useState<NwbData>({ step: "idle", done: false });
52+
53+ useEffect(() => {
54+ if (!file) {
55+ setData({ step: "idle", done: false });
56+ return;
57+ }
58+ let canceled = false;
59+ const update = (patch: Partial<NwbData>) =>
60+ !canceled && setData((d) => ({ ...d, ...patch }));
61+
62+ const run = async () => {
63+ try {
64+ update({ step: "reading file structure (root group)", done: false });
65+ const rootGroup = await file.getGroup("/");
66+ update({ rootGroup });
67+
68+ update({ step: "reading metadata fields" });
69+ const subjectPrefix = "/general/subject";
70+ const meta: Record<string, string> = {
71+ session_description: await readScalar(file, "/session_description"),
72+ identifier: await readScalar(file, "/identifier"),
73+ session_start_time: await readScalar(file, "/session_start_time"),
74+ institution: await readScalar(file, "/general/institution"),
75+ lab: await readScalar(file, "/general/lab"),
76+ subject_id: await readScalar(file, `${subjectPrefix}/subject_id`),
77+ species: await readScalar(file, `${subjectPrefix}/species`),
78+ sex: await readScalar(file, `${subjectPrefix}/sex`),
79+ age: await readScalar(file, `${subjectPrefix}/age`),
80+ };
81+ update({ meta });
82+
83+ update({
84+ step: `lazily reading first ${WINDOW_SAMPLES.toLocaleString()} samples of pupil & running traces`,
85+ });
86+ const pupilDs = await file.getDataset(`${PUPIL}/data`);
87+ const totalSamples = pupilDs?.shape?.[0] ?? 0;
88+ const n = Math.min(WINDOW_SAMPLES, totalSamples || WINDOW_SAMPLES);
89+ const slice: [number, number][] = [[0, n]];
90+ const t = toF64(
91+ await file.getDatasetData(`${PUPIL}/timestamps`, { slice }),
92+ );
93+ const pupil = toF64(
94+ await file.getDatasetData(`${PUPIL}/data`, { slice }),
95+ );
96+ const running = toF64(
97+ await file.getDatasetData(`${RUNNING}/data`, { slice }),
98+ );
99+ update({ timeseries: { t, pupil, running, totalSamples } });
100+
101+ update({ step: "reading trials table (stimulus frequency)" });
102+ const startTime = toF64(
103+ await file.getDatasetData(`${TRIALS}/start_time`, {}),
104+ );
105+ const stimFrequency = toF64(
106+ await file.getDatasetData(`${TRIALS}/stim_frequency`, {}),
107+ );
108+ update({ trials: { startTime, stimFrequency } });
109+
110+ update({ step: "done", done: true });
111+ } catch (err: unknown) {
112+ update({
113+ step: "error",
114+ error: err instanceof Error ? err.message : String(err),
115+ done: true,
116+ });
117+ }
118+ };
119+ run();
120+ return () => {
121+ canceled = true;
122+ };
123+ }, [file]);
124+
125+ return data;
126+};
src/remote-h5-file/.gitignoreadded+1−0View file
@@ -0,0 +1 @@
1+!lib/
src/remote-h5-file/index.tsadded+20−0View file
@@ -0,0 +1,20 @@
1+export {
2+ RemoteH5File,
3+ MergedRemoteH5File,
4+ getRemoteH5File,
5+ getMergedRemoteH5File,
6+ globalRemoteH5FileStats,
7+} from "./lib/RemoteH5File";
8+export type {
9+ RemoteH5FileX,
10+ RemoteH5Dataset,
11+ RemoteH5Group,
12+ RemoteH5Subdataset,
13+ RemoteH5Subgroup,
14+ DatasetDataType,
15+} from "./lib/RemoteH5File";
16+export {
17+ default as RemoteH5FileLindi,
18+ getRemoteH5FileLindi,
19+} from "./lib/lindi/RemoteH5FileLindi";
20+export type { Canceler } from "./lib/helpers";
src/remote-h5-file/lib/RemoteH5File.tsadded+419−0View file
@@ -0,0 +1,419 @@
1+/* eslint-disable @typescript-eslint/no-explicit-any */
2+import { Canceler, postRemoteH5WorkerRequest } from "./helpers";
3+import RemoteH5FileLindi, {
4+ getRemoteH5FileLindi,
5+} from "./lindi/RemoteH5FileLindi";
6+
7+export type RemoteH5FileX =
8+ | RemoteH5File
9+ | MergedRemoteH5File
10+ | RemoteH5FileLindi;
11+
12+export type RemoteH5Group = {
13+ path: string;
14+ subgroups: RemoteH5Subgroup[];
15+ datasets: RemoteH5Subdataset[];
16+ attrs: { [key: string]: any };
17+};
18+
19+export type RemoteH5Subgroup = {
20+ name: string;
21+ path: string;
22+ attrs: { [key: string]: any };
23+};
24+
25+export type RemoteH5Subdataset = {
26+ name: string;
27+ path: string;
28+ shape: number[];
29+ dtype: string;
30+ attrs: { [key: string]: any };
31+ chunks?: number[];
32+ compressor?: string;
33+ filters?: string[];
34+};
35+
36+export type RemoteH5Dataset = {
37+ name: string;
38+ path: string;
39+ shape: number[];
40+ dtype: string;
41+ attrs: { [key: string]: any };
42+ chunks?: number[];
43+ compressor?: string;
44+ filters?: string[];
45+};
46+
47+export type DatasetDataType =
48+ | Float32Array
49+ | Float64Array
50+ | Int8Array
51+ | Int16Array
52+ | Int32Array
53+ | Uint8Array
54+ | Uint16Array
55+ | Uint32Array;
56+
57+const defaultChunkSize = 1024 * 100;
58+// const defaultChunkSize = 1024 * 1024 * 2
59+
60+export const globalRemoteH5FileStats = {
61+ getGroupCount: 0,
62+ getDatasetCount: 0,
63+ getDatasetDataCount: 0,
64+ numPendingRequests: 0,
65+};
66+
67+type GetGroupResponse = {
68+ success: boolean;
69+ group?: RemoteH5Group;
70+};
71+
72+type GetDatasetResponse = {
73+ success: boolean;
74+ dataset?: RemoteH5Dataset;
75+};
76+
77+export class RemoteH5File {
78+ #groupCache: { [path: string]: GetGroupResponse | null } = {}; // null means in progress
79+ #datasetCache: { [path: string]: GetDatasetResponse | null } = {}; // null means in progress
80+ #sourceUrls: string[] | undefined = undefined;
81+ constructor(
82+ public url: string,
83+ private o: { chunkSize?: number },
84+ ) {}
85+ get dataIsRemote() {
86+ return !this.url.startsWith("http://localhost");
87+ }
88+ async getGroup(path: string): Promise<RemoteH5Group | undefined> {
89+ const cc = this.#groupCache[path];
90+ if (cc) return cc.group;
91+ if (cc === null) {
92+ // in progress
93+ while (this.#groupCache[path] === null) {
94+ await new Promise((resolve) => setTimeout(resolve, 100));
95+ }
96+ const cc2 = this.#groupCache[path];
97+ if (cc2) return cc2.group;
98+ else throw Error("Unexpected");
99+ }
100+ this.#groupCache[path] = null;
101+ const dummyCanceler = { onCancel: [] };
102+ let resp;
103+ try {
104+ resp = await postRemoteH5WorkerRequest(
105+ {
106+ type: "getGroup",
107+ url: this.url,
108+ path,
109+ chunkSize: this.o.chunkSize || defaultChunkSize,
110+ },
111+ dummyCanceler,
112+ );
113+ } catch {
114+ this.#groupCache[path] = { success: false };
115+ return undefined;
116+ }
117+ this.#groupCache[path] = resp;
118+ globalRemoteH5FileStats.getGroupCount++;
119+ return resp.group;
120+ }
121+ async getDataset(path: string): Promise<RemoteH5Dataset | undefined> {
122+ const cc = this.#datasetCache[path];
123+ if (cc) return cc.dataset;
124+ if (cc === null) {
125+ // in progress
126+ while (this.#datasetCache[path] === null) {
127+ await new Promise((resolve) => setTimeout(resolve, 100));
128+ }
129+ const cc2 = this.#datasetCache[path];
130+ if (cc2) return cc2.dataset;
131+ else throw Error("Unexpected");
132+ }
133+ this.#datasetCache[path] = null;
134+ const dummyCanceler = { onCancel: [] };
135+ let resp;
136+ try {
137+ resp = await postRemoteH5WorkerRequest(
138+ {
139+ type: "getDataset",
140+ url: this.url,
141+ path,
142+ chunkSize: this.o.chunkSize || defaultChunkSize,
143+ },
144+ dummyCanceler,
145+ );
146+ } catch {
147+ this.#datasetCache[path] = { success: false };
148+ return undefined;
149+ }
150+ this.#datasetCache[path] = resp;
151+ globalRemoteH5FileStats.getDatasetCount++;
152+ return resp.dataset;
153+ }
154+ async getDatasetData(
155+ path: string,
156+ o: {
157+ slice?: [number, number][];
158+ allowBigInt?: boolean;
159+ canceler?: Canceler;
160+ },
161+ ): Promise<DatasetDataType | undefined> {
162+ if (o.slice) {
163+ for (const ss of o.slice) {
164+ if (isNaN(ss[0]) || isNaN(ss[1])) {
165+ console.warn("Invalid slice", path, o.slice);
166+ throw Error("Invalid slice");
167+ }
168+ }
169+ }
170+ const ds = await this.getDataset(path);
171+ if (!ds) return undefined;
172+ let urlToUse: string = this.url;
173+ if (product(ds.shape) > 100) {
174+ urlToUse = this.url;
175+ }
176+
177+ const { slice, allowBigInt, canceler } = o;
178+ const dummyCanceler = { onCancel: [] };
179+ let resp;
180+ try {
181+ resp = await postRemoteH5WorkerRequest(
182+ {
183+ type: "getDatasetData",
184+ url: urlToUse,
185+ path,
186+ slice,
187+ chunkSize: this.o.chunkSize || defaultChunkSize,
188+ },
189+ canceler || dummyCanceler,
190+ );
191+ } catch {
192+ return undefined;
193+ }
194+ const { data } = resp;
195+ let x = data;
196+ if (!allowBigInt) {
197+ // check if x is a BigInt64Array
198+ if (x && x.constructor && x.constructor.name === "BigInt64Array") {
199+ // convert to Int32Array
200+ const y = new Int32Array(x.length);
201+ for (let i = 0; i < x.length; i++) {
202+ y[i] = Number(x[i]);
203+ }
204+ x = y;
205+ }
206+ // check if x is a BigUint64Array
207+ if (x && x.constructor && x.constructor.name === "BigUint64Array") {
208+ // convert to Uint32Array
209+ const y = new Uint32Array(x.length);
210+ for (let i = 0; i < x.length; i++) {
211+ y[i] = Number(x[i]);
212+ }
213+ x = y;
214+ }
215+ }
216+ globalRemoteH5FileStats.getDatasetDataCount++;
217+ return x;
218+ }
219+ getUrls() {
220+ return [this.url];
221+ }
222+ get sourceUrls(): string[] | undefined {
223+ return this.#sourceUrls;
224+ }
225+ set sourceUrls(v: string[] | undefined) {
226+ this.#sourceUrls = v;
227+ }
228+}
229+
230+export class MergedRemoteH5File {
231+ #files: RemoteH5FileX[];
232+ #sourceUrls: string[] | undefined = undefined;
233+ constructor(files: RemoteH5FileX[]) {
234+ this.#files = files;
235+ }
236+ get dataIsRemote() {
237+ return this.#files.some((f) => {
238+ if (f instanceof RemoteH5File) {
239+ return f.dataIsRemote;
240+ } else if (f instanceof MergedRemoteH5File) {
241+ // this case shouldn't happen - unfortunately we can't call f.dataIsRemote here because typescript doesn't allow it
242+ return false;
243+ } else {
244+ throw Error("Unexpected");
245+ }
246+ });
247+ }
248+ async getGroup(path: string): Promise<RemoteH5Group | undefined> {
249+ const allGroups: RemoteH5Group[] = [];
250+ for (const f of this.#files) {
251+ const gg = await f.getGroup(path);
252+ if (gg) allGroups.push(gg);
253+ }
254+ console.log(`Got ${allGroups.length} groups`, path);
255+ if (allGroups.length === 0) return undefined;
256+ const ret = mergeGroups(allGroups);
257+ return ret;
258+ }
259+ async getDataset(path: string): Promise<RemoteH5Dataset | undefined> {
260+ for (const f of this.#files) {
261+ const dd = await f.getDataset(path);
262+ if (dd) {
263+ // just return the first one
264+ return dd;
265+ }
266+ }
267+ return undefined;
268+ }
269+ async getDatasetData(
270+ path: string,
271+ o: {
272+ slice?: [number, number][];
273+ allowBigInt?: boolean;
274+ canceler?: Canceler;
275+ },
276+ ): Promise<DatasetDataType | undefined> {
277+ let canceled = false;
278+ o.canceler?.onCancel.push(() => {
279+ canceled = true;
280+ });
281+ for (const f of this.#files) {
282+ const dd = await f.getDatasetData(path, o);
283+ if (dd) {
284+ // just return the first one
285+ return dd;
286+ }
287+ if (canceled) return undefined;
288+ }
289+ return undefined;
290+ }
291+ getFiles() {
292+ return this.#files;
293+ }
294+ getUrls(): string[] {
295+ return this.#files.flatMap((f) => f.getUrls());
296+ }
297+ get sourceUrls(): string[] | undefined {
298+ return this.#sourceUrls;
299+ }
300+ set sourceUrls(v: string[] | undefined) {
301+ this.#sourceUrls = v;
302+ }
303+}
304+
305+const mergeGroups = (groups: RemoteH5Group[]): RemoteH5Group => {
306+ if (groups.length === 0) throw Error("Unexpected groups.length == 0");
307+ const ret: RemoteH5Group = {
308+ path: groups[0].path,
309+ subgroups: [],
310+ datasets: [],
311+ attrs: {},
312+ };
313+ const allSubgroupNames: string[] = [];
314+ const allDatasetNames: string[] = [];
315+ for (const g of groups) {
316+ for (const sg of g.subgroups) {
317+ if (!allSubgroupNames.includes(sg.name)) {
318+ allSubgroupNames.push(sg.name);
319+ }
320+ }
321+ for (const ds of g.datasets) {
322+ if (!allDatasetNames.includes(ds.name)) {
323+ allDatasetNames.push(ds.name);
324+ }
325+ }
326+ }
327+ for (const sgName of allSubgroupNames) {
328+ const subgroups: RemoteH5Subgroup[] = [];
329+ for (const g of groups) {
330+ const sg = g.subgroups.find((s) => s.name === sgName);
331+ if (sg) subgroups.push(sg);
332+ }
333+ ret.subgroups.push(mergeSubgroups(subgroups));
334+ }
335+ for (const dsName of allDatasetNames) {
336+ const datasets: RemoteH5Subdataset[] = [];
337+ for (const g of groups) {
338+ const ds = g.datasets.find((d) => d.name === dsName);
339+ if (ds) datasets.push(ds);
340+ }
341+ // for the datasets we just use the first one
342+ if (datasets.length > 0) {
343+ ret.datasets.push(datasets[0]);
344+ }
345+ }
346+ for (const g of groups) {
347+ for (const key in g.attrs) {
348+ if (!(key in ret.attrs)) {
349+ // the first takes precedence
350+ ret.attrs[key] = g.attrs[key];
351+ }
352+ }
353+ }
354+ return ret;
355+};
356+
357+const mergeSubgroups = (subgroups: RemoteH5Subgroup[]): RemoteH5Subgroup => {
358+ if (subgroups.length === 0) throw Error("Unexpected subgroups.length == 0");
359+ const ret: RemoteH5Subgroup = {
360+ name: subgroups[0].name,
361+ path: subgroups[0].path,
362+ attrs: {},
363+ };
364+ for (const g of subgroups) {
365+ for (const key in g.attrs) {
366+ if (!(key in ret.attrs)) {
367+ // the first takes precedence
368+ ret.attrs[key] = g.attrs[key];
369+ }
370+ }
371+ }
372+ return ret;
373+};
374+
375+const globalRemoteH5Files: { [url: string]: RemoteH5File } = {};
376+export const getRemoteH5File = async (url: string) => {
377+ const kk = url;
378+ if (!globalRemoteH5Files[kk]) {
379+ globalRemoteH5Files[kk] = new RemoteH5File(url, {});
380+ }
381+ return globalRemoteH5Files[kk];
382+};
383+
384+const globalMergedRemoteH5Files: { [kk: string]: MergedRemoteH5File } = {};
385+export const getMergedRemoteH5File = async (
386+ urls: string[],
387+ storageType: ("h5" | "zarr" | "lindi")[],
388+) => {
389+ if (urls.length === 0) throw Error(`Length of urls must be > 0`);
390+ if (storageType.length !== urls.length)
391+ throw Error(`Length of storageType must be equal to length of urls`);
392+ if (urls.length === 1) {
393+ if (storageType[0] === "lindi") {
394+ return await getRemoteH5FileLindi(urls[0]);
395+ } else {
396+ return await getRemoteH5File(urls[0]);
397+ }
398+ }
399+ const kk = urls.join("|");
400+ if (!globalMergedRemoteH5Files[kk]) {
401+ const files = await Promise.all(
402+ urls.map((url, i) => {
403+ if (storageType[i] === "lindi") {
404+ return getRemoteH5FileLindi(url);
405+ } else {
406+ return getRemoteH5File(url);
407+ }
408+ }),
409+ );
410+ globalMergedRemoteH5Files[kk] = new MergedRemoteH5File(files);
411+ }
412+ return globalMergedRemoteH5Files[kk];
413+};
414+
415+const product = (x: number[]) => {
416+ let p = 1;
417+ for (let i = 0; i < x.length; i++) p *= x[i];
418+ return p;
419+};
src/remote-h5-file/lib/helpers.tsadded+148−0View file
@@ -0,0 +1,148 @@
1+/* eslint-disable @typescript-eslint/no-explicit-any */
2+/* eslint-disable @typescript-eslint/no-non-null-assertion */
3+
4+import { globalRemoteH5FileStats } from "./RemoteH5File";
5+
6+type RRequest = {
7+ requestId: string;
8+ request: any;
9+ onResolved: (resp: any) => void;
10+ onRejected: (err: Error) => void;
11+};
12+
13+export type Canceler = { onCancel: (() => void)[] };
14+
15+// Returns a blob:// URL which points
16+// to a javascript file which will call
17+// importScripts with the given URL
18+// See: https://stackoverflow.com/a/62914052
19+const getWorkerURL = (url: string) => {
20+ const content = `importScripts( "${url}" );`;
21+ return URL.createObjectURL(new Blob([content], { type: "text/javascript" }));
22+};
23+
24+const createWorker = (url: string) => {
25+ const workerUrl = getWorkerURL(url);
26+ return new Worker(workerUrl);
27+};
28+
29+class RemoteH5WorkerWrapper {
30+ #worker: Worker;
31+ #pendingRequests: RRequest[] = [];
32+ #runningRequest: RRequest | undefined = undefined;
33+ constructor() {
34+ // this.#worker = new Worker(new URL('./RemoteH5Worker.js', import.meta.url), { type: 'module' })
35+
36+ // here's the source of truth:
37+ // this.#worker = createWorker('https://cdn.jsdelivr.net/gh/magland/remote-h5-worker@0.1.2/dist/RemoteH5Worker.js');
38+
39+ // but maybe it's faster and more reliable to load from cloudflare
40+ this.#worker = createWorker("https://tempory.net/js/RemoteH5Worker.js");
41+ }
42+ get numRunningRequests() {
43+ return this.#runningRequest ? 1 : 0;
44+ }
45+ get numPendingRequests() {
46+ return this.#pendingRequests.length;
47+ }
48+ get numRequests() {
49+ return this.numRunningRequests + this.numPendingRequests;
50+ }
51+ async postRequest(req: any, canceler: Canceler) {
52+ const requestId = Math.random().toString();
53+ const resp = await new Promise<any>((resolve, reject) => {
54+ this.#pendingRequests.push({
55+ requestId,
56+ request: req,
57+ onResolved: resolve,
58+ onRejected: reject,
59+ });
60+ canceler.onCancel.push(() => {
61+ const ind = this.#pendingRequests.findIndex(
62+ (rr) => rr.requestId === requestId,
63+ );
64+ if (ind >= 0) {
65+ this.#pendingRequests.splice(ind, 1);
66+ reject(new Error("canceled"));
67+ }
68+ });
69+ this._processPendingRequests();
70+ });
71+ return resp;
72+ }
73+ _processPendingRequests = () => {
74+ if (this.#runningRequest) return;
75+ if (this.#pendingRequests.length === 0) return;
76+ const rr = this.#pendingRequests.shift()!;
77+ this.#runningRequest = rr;
78+ let completed = false;
79+ const doResolve = (resp: any) => {
80+ if (completed) return;
81+ completed = true;
82+ this.#worker.removeEventListener("message", listener);
83+ rr.onResolved(resp);
84+ this.#runningRequest = undefined;
85+ this._processPendingRequests();
86+ };
87+ const doReject = (err: Error) => {
88+ if (completed) return;
89+ completed = true;
90+ this.#worker.removeEventListener("message", listener);
91+ rr.onRejected(err);
92+ this.#runningRequest = undefined;
93+ this._processPendingRequests();
94+ };
95+ const listener = (e: MessageEvent) => {
96+ const d = e.data;
97+ if (d.type === "response" && d.requestId === rr.requestId) {
98+ if (d.response.success) {
99+ doResolve(d.response);
100+ } else {
101+ doReject(new Error(d.response.error));
102+ }
103+ }
104+ };
105+ this.#worker.addEventListener("message", listener);
106+ this.#worker.postMessage({
107+ type: "request",
108+ requestId: rr.requestId,
109+ request: rr.request,
110+ });
111+ setTimeout(() => {
112+ doReject(new Error("timeout"));
113+ }, 60000 * 3);
114+ };
115+}
116+
117+// While it sounds like a good idea to have a lot of workers (for concurrent http requests), there is a problem
118+// in that each worker needs to load the meta information for the hdf5 file... which takes some time
119+// therefore too many workers => initial slowdown
120+const numWorkers = 1;
121+class RemoteH5WorkerManager {
122+ #workers: RemoteH5WorkerWrapper[] = [];
123+ constructor() {
124+ for (let i = 0; i < numWorkers; i++) {
125+ this.#workers.push(new RemoteH5WorkerWrapper());
126+ }
127+ }
128+ async postRequest(req: any, canceler: Canceler) {
129+ const worker = this.#workers.sort(
130+ (a, b) => a.numRequests - b.numRequests,
131+ )[0];
132+ return await worker.postRequest(req, canceler);
133+ }
134+}
135+const workerManager = new RemoteH5WorkerManager();
136+
137+export const postRemoteH5WorkerRequest = async (
138+ req: any,
139+ canceler: Canceler,
140+) => {
141+ globalRemoteH5FileStats.numPendingRequests++;
142+ try {
143+ const ret = await workerManager.postRequest(req, canceler);
144+ return ret;
145+ } finally {
146+ globalRemoteH5FileStats.numPendingRequests--;
147+ }
148+};
src/remote-h5-file/lib/lindi/ReferenceFileSystemClient.tsadded+236−0View file
@@ -0,0 +1,236 @@
1+// @ts-nocheck
2+import { ZMetaDataZArray } from "./RemoteH5FileLindi";
3+import zarrDecodeChunkArray from "./zarrDecodeChunkArray";
4+
5+/* eslint-disable @typescript-eslint/no-explicit-any */
6+export type ReferenceFileSystemObject = {
7+ version?: any;
8+ refs: { [key: string]: string | [string, number, number] };
9+ templates?: { [key: string]: string };
10+};
11+
12+export const isReferenceFileSystemObject = (
13+ x: any,
14+): x is ReferenceFileSystemObject => {
15+ if (!x) return false;
16+ if (typeof x !== "object") return false;
17+ if (!x.refs) return false;
18+ return true;
19+};
20+
21+export interface RemoteTarInterface {
22+ url: string;
23+ getByteRangeForFile: (
24+ fileName: string,
25+ ) => Promise<{ startByte: number; endByte: number }>;
26+}
27+
28+export class ReferenceFileSystemClient {
29+ #fileContentCache: {
30+ [key: string]: { content: any | undefined; found: boolean };
31+ } = {};
32+ #inProgressReads: { [key: string]: boolean } = {};
33+ constructor(
34+ private obj: ReferenceFileSystemObject,
35+ private remoteTar: RemoteTarInterface | undefined,
36+ ) {}
37+ async readJson(path: string): Promise<{ [key: string]: any } | undefined> {
38+ const buf = await this.readBinary(path, { decodeArray: false });
39+ if (!buf) return undefined;
40+ const text = new TextDecoder().decode(buf);
41+ // replace NaN by "NaN" so that JSON.parse doesn't choke on it
42+ // text = text.replace(/NaN/g, '"___NaN___"'); // This is not ideal. See: https://stackoverflow.com/a/15228712
43+ // BUT we want to make sure we don't replace NaN within quoted strings
44+ // Here's an example where this matters: https://neurosift.app/?p=/nwb&dandisetId=000409&dandisetVersion=draft&url=https://api.dandiarchive.org/api/assets/54b277ce-2da7-4730-b86b-cfc8dbf9c6fd/download/
45+ // raw/intervals/contrast_left
46+ let newText: string;
47+ if (text.includes("NaN")) {
48+ newText = "";
49+ let inString = false;
50+ let isEscaped = false;
51+ for (let i = 0; i < text.length; i++) {
52+ const c = text[i];
53+ if (c === '"' && !isEscaped) inString = !inString;
54+ if (!inString && c === "N" && text.slice(i, i + 3) === "NaN") {
55+ newText += '"___NaN___"';
56+ i += 2;
57+ } else {
58+ newText += c;
59+ }
60+ isEscaped = c === "\\" && !isEscaped;
61+ }
62+ } else {
63+ newText = text;
64+ }
65+ try {
66+ return JSON.parse(newText, (_key, value) => {
67+ if (value === "___NaN___") return NaN;
68+ return value;
69+ });
70+ } catch (e) {
71+ console.warn(text);
72+ throw Error("Failed to parse JSON for " + path + ": " + e);
73+ }
74+ }
75+ async readBinary(
76+ path: string,
77+ o: {
78+ decodeArray?: boolean;
79+ startByte?: number;
80+ endByte?: number;
81+ disableCache?: boolean;
82+ },
83+ ): Promise<any | undefined> {
84+ if (o.startByte !== undefined) {
85+ if (o.decodeArray)
86+ throw Error("Cannot decode array and read a slice at the same time");
87+ if (o.endByte === undefined)
88+ throw Error("If you specify startByte, you must also specify endByte");
89+ } else if (o.endByte !== undefined) {
90+ throw Error("If you specify endByte, you must also specify startByte");
91+ }
92+ if (
93+ o.endByte !== undefined &&
94+ o.startByte !== undefined &&
95+ o.endByte < o.startByte
96+ ) {
97+ throw Error(
98+ `endByte must be greater than or equal to startByte: ${o.startByte} ${o.endByte} for ${path}`,
99+ );
100+ }
101+ if (
102+ o.endByte !== undefined &&
103+ o.startByte !== undefined &&
104+ o.endByte === o.startByte
105+ ) {
106+ return new ArrayBuffer(0);
107+ }
108+ const kk =
109+ path +
110+ "|" +
111+ (o.decodeArray ? "decode" : "") +
112+ "|" +
113+ o.startByte +
114+ "|" +
115+ o.endByte;
116+ while (this.#inProgressReads[kk]) {
117+ await new Promise((resolve) => setTimeout(resolve, 100));
118+ }
119+ this.#inProgressReads[kk] = true;
120+ try {
121+ if (path.startsWith("/")) path = path.slice(1);
122+ if (this.#fileContentCache[kk]) {
123+ if (this.#fileContentCache[kk].found) {
124+ return this.#fileContentCache[kk].content;
125+ }
126+ return undefined;
127+ }
128+ const ref = this.obj.refs[path];
129+ if (!ref) return undefined;
130+ let buf: ArrayBuffer | undefined;
131+ if (typeof ref === "string") {
132+ if (ref.startsWith("base64:")) {
133+ buf = _base64ToArrayBuffer(ref.slice("base64:".length));
134+ } else {
135+ // just a string
136+ buf = new TextEncoder().encode(ref).buffer;
137+ }
138+ if (o.startByte !== undefined) {
139+ buf = buf.slice(o.startByte, o.endByte);
140+ }
141+ } else if (typeof ref === "object" && Array.isArray(ref)) {
142+ if (ref.length !== 3) throw Error(`Invalid ref for ${path}`);
143+ let refUrl = this._applyTemplates(ref[0]);
144+ let start = ref[1];
145+ let numBytes = ref[2];
146+ if (refUrl.startsWith("./") && this.remoteTar) {
147+ const { startByte } = await this._getByteRangeForFileInRemoteTar(
148+ refUrl.slice("./".length),
149+ );
150+ refUrl = this.remoteTar.url;
151+ start = startByte + start;
152+ }
153+ if (o.startByte !== undefined) {
154+ start += o.startByte;
155+ numBytes = o.endByte! - o.startByte;
156+ }
157+ let url0 = refUrl;
158+ if (o.disableCache) {
159+ url0 += `?cacheBust=${Date.now()}`;
160+ }
161+ const r = await fetch(url0, {
162+ headers: {
163+ Range: `bytes=${start}-${start + numBytes - 1}`,
164+ },
165+ });
166+ if (!r.ok) throw Error("Failed to fetch " + refUrl);
167+ buf = await r.arrayBuffer();
168+ } else if (typeof ref === "object") {
169+ buf = new TextEncoder().encode(JSON.stringify(ref)).buffer;
170+ } else {
171+ throw Error("Invalid ref for " + path);
172+ }
173+ if (o.decodeArray) {
174+ const parentPath = path.split("/").slice(0, -1).join("/");
175+ const zarray = (await this.readJson(parentPath + "/.zarray")) as
176+ | ZMetaDataZArray
177+ | undefined;
178+ if (!zarray) throw Error("Failed to read .zarray for " + path);
179+ try {
180+ buf = await zarrDecodeChunkArray(
181+ buf,
182+ zarray.dtype,
183+ zarray.compressor,
184+ zarray.filters,
185+ zarray.chunks,
186+ );
187+ } catch (e) {
188+ throw Error(`Failed to decode chunk array for ${path}: ${e}`);
189+ }
190+ }
191+ if (buf) {
192+ this.#fileContentCache[kk] = { content: buf, found: true };
193+ } else {
194+ this.#fileContentCache[kk] = { content: undefined, found: false };
195+ }
196+ return buf;
197+ } catch (e) {
198+ this.#fileContentCache[kk] = { content: undefined, found: false }; // important to do this so we don't keep trying to read the same file
199+ throw e;
200+ } finally {
201+ this.#inProgressReads[kk] = false;
202+ }
203+ }
204+ get _refs() {
205+ return this.obj.refs;
206+ }
207+ _applyTemplates(s: string): string {
208+ if (s.includes("{{") && s.includes("}}") && this.obj.templates) {
209+ for (const [k, v] of Object.entries(this.obj.templates)) {
210+ s = s.replace("{{" + k + "}}", v);
211+ }
212+ return s;
213+ } else {
214+ return s;
215+ }
216+ }
217+ private async _getByteRangeForFileInRemoteTar(fileName: string) {
218+ if (!this.remoteTar) {
219+ throw Error("Unexpected");
220+ }
221+ const { startByte, endByte } =
222+ await this.remoteTar.getByteRangeForFile(fileName);
223+ return { startByte, endByte };
224+ }
225+}
226+
227+function _base64ToArrayBuffer(base64: string) {
228+ const binary_string = window.atob(base64);
229+ const bytes = new Uint8Array(binary_string.length);
230+ for (let i = 0; i < binary_string.length; i++) {
231+ bytes[i] = binary_string.charCodeAt(i);
232+ }
233+ return bytes;
234+}
235+
236+export default ReferenceFileSystemClient;
src/remote-h5-file/lib/lindi/RemoteH5FileLindi.tsadded+598−0View file
@@ -0,0 +1,598 @@
1+/* eslint-disable @typescript-eslint/no-explicit-any */
2+import {
3+ DatasetDataType,
4+ RemoteH5Dataset,
5+ RemoteH5Group,
6+ RemoteH5Subdataset,
7+ RemoteH5Subgroup,
8+ // getRemoteH5File,
9+ globalRemoteH5FileStats,
10+} from "../RemoteH5File";
11+// import { Canceler } from "../helpers";
12+
13+type Canceler = {
14+ onCancel: (() => void)[];
15+};
16+
17+import ReferenceFileSystemClient, {
18+ ReferenceFileSystemObject,
19+ RemoteTarInterface,
20+ isReferenceFileSystemObject,
21+} from "./ReferenceFileSystemClient";
22+import lindiDatasetDataLoader from "./lindiDatasetDataLoader";
23+import zarrDecodeChunkArray from "./zarrDecodeChunkArray";
24+
25+type ZMetaDataZAttrs = { [key: string]: any };
26+
27+type ZMetaDataZGroup = {
28+ zarr_format: number;
29+};
30+
31+export type ZMetaDataZArray = {
32+ chunks?: number[];
33+ compressor?: any;
34+ dtype?: string;
35+ fill_value?: any;
36+ filters?: any[];
37+ order?: "C" | "F";
38+ shape?: number[];
39+ zarr_format?: 2;
40+};
41+
42+export class ZarrFileSystemClient {
43+ #fileContentCache: {
44+ [key: string]: { content: any | undefined; found: boolean };
45+ } = {};
46+ #inProgressReads: { [key: string]: boolean } = {};
47+ constructor(
48+ private url: string,
49+ private zmetadata: any,
50+ ) {}
51+ async readJson(path: string): Promise<{ [key: string]: any } | undefined> {
52+ if (path in this.zmetadata.metadata) {
53+ return this.zmetadata.metadata[path];
54+ }
55+ const lastPartOfPath = path.split("/").slice(-1)[0];
56+ if (lastPartOfPath.startsWith(".")) {
57+ // if it's not in the metadata, we assume it's not there
58+ return undefined;
59+ }
60+ const buf = await this.readBinary(path, { decodeArray: false });
61+ if (!buf) return undefined;
62+ const text = new TextDecoder().decode(buf);
63+ try {
64+ return JSON.parse(text, (_key, value) => {
65+ if (value === "___NaN___") return NaN;
66+ return value;
67+ });
68+ } catch (e) {
69+ console.warn(text);
70+ throw Error("Failed to parse JSON for " + path + ": " + e);
71+ }
72+ }
73+ async readBinary(
74+ path: string,
75+ o: {
76+ decodeArray?: boolean;
77+ startByte?: number;
78+ endByte?: number;
79+ disableCache?: boolean;
80+ },
81+ ): Promise<any | undefined> {
82+ if (o.startByte !== undefined) {
83+ if (o.decodeArray)
84+ throw Error("Cannot decode array and read a slice at the same time");
85+ if (o.endByte === undefined)
86+ throw Error("If you specify startByte, you must also specify endByte");
87+ } else if (o.endByte !== undefined) {
88+ throw Error("If you specify endByte, you must also specify startByte");
89+ }
90+ if (
91+ o.endByte !== undefined &&
92+ o.startByte !== undefined &&
93+ o.endByte < o.startByte
94+ ) {
95+ throw Error(
96+ `endByte must be greater than or equal to startByte: ${o.startByte} ${o.endByte} for ${path}`,
97+ );
98+ }
99+ if (
100+ o.endByte !== undefined &&
101+ o.startByte !== undefined &&
102+ o.endByte === o.startByte
103+ ) {
104+ return new ArrayBuffer(0);
105+ }
106+ const kk =
107+ path +
108+ "|" +
109+ (o.decodeArray ? "decode" : "") +
110+ "|" +
111+ o.startByte +
112+ "|" +
113+ o.endByte;
114+ while (this.#inProgressReads[kk]) {
115+ await new Promise((resolve) => setTimeout(resolve, 100));
116+ }
117+ this.#inProgressReads[kk] = true;
118+ try {
119+ if (path.startsWith("/")) path = path.slice(1);
120+ if (this.#fileContentCache[kk]) {
121+ if (this.#fileContentCache[kk].found) {
122+ return this.#fileContentCache[kk].content;
123+ }
124+ return undefined;
125+ }
126+ const url = this.url + "/" + path;
127+ let buf: ArrayBuffer | undefined;
128+ if (o.startByte !== undefined && o.endByte !== undefined) {
129+ buf = await fetchByteRange(url, o.startByte, o.endByte - o.startByte);
130+ } else {
131+ const r = await fetch(url);
132+ if (!r.ok) {
133+ if (r.status === 404) {
134+ this.#fileContentCache[kk] = { content: undefined, found: false };
135+ return undefined; // file not found
136+ }
137+ throw Error(`Failed to fetch ${url}: ${r.statusText}`);
138+ }
139+ buf = await r.arrayBuffer();
140+ }
141+ if (o.decodeArray) {
142+ const parentPath = path.split("/").slice(0, -1).join("/");
143+ const zarray = (await this.readJson(parentPath + "/.zarray")) as
144+ | ZMetaDataZArray
145+ | undefined;
146+ if (!zarray) throw Error("Failed to read .zarray for " + path);
147+ try {
148+ buf = await zarrDecodeChunkArray(
149+ buf,
150+ zarray.dtype,
151+ zarray.compressor,
152+ zarray.filters,
153+ zarray.chunks,
154+ );
155+ } catch (e) {
156+ throw Error(`Failed to decode chunk array for ${path}: ${e}`);
157+ }
158+ }
159+ if (buf) {
160+ this.#fileContentCache[kk] = { content: buf, found: true };
161+ } else {
162+ this.#fileContentCache[kk] = { content: undefined, found: false };
163+ }
164+ return buf;
165+ } catch (e) {
166+ this.#fileContentCache[kk] = { content: undefined, found: false }; // important to do this so we don't keep trying to read the same file
167+ throw e;
168+ } finally {
169+ this.#inProgressReads[kk] = false;
170+ }
171+ }
172+}
173+
174+class RemoteH5FileLindi {
175+ #cacheDisabled = false; // just for benchmarking
176+ #sourceUrls: string[] | undefined = undefined;
177+ constructor(
178+ public url: string,
179+ private lindiFileSystemClient:
180+ | ReferenceFileSystemClient
181+ | ZarrFileSystemClient,
182+ private pathsByParentPath: { [key: string]: string[] },
183+ ) {}
184+ static async create(url: string) {
185+ const { rfs: obj, remoteTar } = await fetchRfsFromRemoteLindi(url);
186+ // console.info(`reference file system for ${url}`, obj);
187+ // console.info(`Meta only`, metaOnly(obj));
188+ const pathsByParentPath: { [key: string]: string[] } = {};
189+ for (const path in obj.refs) {
190+ if (path === ".zattrs" || path === ".zgroup") continue;
191+ const parts = path.split("/");
192+ if (parts.length <= 1) continue;
193+ const lastPart = parts[parts.length - 1];
194+ if (
195+ lastPart === ".zattrs" ||
196+ lastPart === ".zgroup" ||
197+ lastPart === ".zarray"
198+ ) {
199+ const thePath = parts.slice(0, parts.length - 1).join("/");
200+ const theParentPath = parts.slice(0, parts.length - 2).join("/");
201+ if (!pathsByParentPath[theParentPath])
202+ pathsByParentPath[theParentPath] = [];
203+ if (!pathsByParentPath[theParentPath].includes(thePath)) {
204+ pathsByParentPath[theParentPath].push(thePath);
205+ }
206+ }
207+ }
208+ return new RemoteH5FileLindi(
209+ url,
210+ new ReferenceFileSystemClient(obj, remoteTar),
211+ pathsByParentPath,
212+ );
213+ }
214+ static async createFromZarr(url: string) {
215+ const zmetadataUrl = `${url}/.zmetadata`;
216+ const zmetadataResponse = await fetch(zmetadataUrl);
217+ if (!zmetadataResponse.ok) {
218+ throw new Error(`Failed to fetch Zarr metadata from ${zmetadataUrl}`);
219+ }
220+ const zmetadata = await zmetadataResponse.json();
221+ const zarrFileSystemClient = new ZarrFileSystemClient(url, zmetadata);
222+ return new RemoteH5FileLindi(url, zarrFileSystemClient, {});
223+ }
224+ get dataIsRemote() {
225+ return !this.url.startsWith("http://localhost");
226+ }
227+ async getGroup(path: string): Promise<RemoteH5Group | undefined> {
228+ if (path === "") path = "/";
229+ let group: RemoteH5Group | undefined;
230+ const pathWithoutBeginningSlash = path.startsWith("/")
231+ ? path.slice(1)
232+ : path;
233+ let zgroup: ZMetaDataZGroup | undefined;
234+ let zattrs: ZMetaDataZAttrs | undefined;
235+ if (path === "/") {
236+ zgroup = (await this.lindiFileSystemClient.readJson(".zgroup")) as
237+ | ZMetaDataZGroup
238+ | undefined;
239+ zattrs = (await this.lindiFileSystemClient.readJson(".zattrs")) as
240+ | ZMetaDataZAttrs
241+ | undefined;
242+ } else {
243+ zgroup = (await this.lindiFileSystemClient.readJson(
244+ pathWithoutBeginningSlash + "/.zgroup",
245+ )) as ZMetaDataZGroup | undefined;
246+ zattrs = (await this.lindiFileSystemClient.readJson(
247+ pathWithoutBeginningSlash + "/.zattrs",
248+ )) as ZMetaDataZAttrs | undefined;
249+ }
250+ if (zgroup) {
251+ const subgroups: RemoteH5Subgroup[] = [];
252+ const subdatasets: RemoteH5Subdataset[] = [];
253+ const childPaths: string[] =
254+ this.pathsByParentPath[pathWithoutBeginningSlash] || [];
255+ for (const childPath of childPaths) {
256+ const childZgroup = await this.lindiFileSystemClient.readJson(
257+ childPath + "/.zgroup",
258+ );
259+ const childZarray = await this.lindiFileSystemClient.readJson(
260+ childPath + "/.zarray",
261+ );
262+ const childZattrs = await this.lindiFileSystemClient.readJson(
263+ childPath + "/.zattrs",
264+ );
265+ if (childZgroup) {
266+ subgroups.push({
267+ name: getNameFromPath(childPath),
268+ path: "/" + childPath,
269+ attrs: childZattrs || {},
270+ });
271+ } else if (childZarray) {
272+ const shape = childZarray.shape;
273+ const dtype = childZarray.dtype;
274+ if (shape && dtype) {
275+ subdatasets.push({
276+ name: getNameFromPath(childPath),
277+ path: "/" + childPath,
278+ shape,
279+ dtype,
280+ attrs: childZattrs || {},
281+ chunks: childZarray.chunks,
282+ compressor: formatCompressor(childZarray.compressor),
283+ filters: formatFilters(childZarray.filters),
284+ });
285+ } else {
286+ console.warn("Unexpected .zarray item", childPath, childZarray);
287+ }
288+ }
289+ }
290+ group = {
291+ path: path,
292+ subgroups,
293+ datasets: subdatasets,
294+ attrs: zattrs || {},
295+ };
296+ }
297+ globalRemoteH5FileStats.getGroupCount++;
298+ return group;
299+ }
300+ async getDataset(path: string): Promise<RemoteH5Dataset | undefined> {
301+ const pathWithoutBeginningSlash = path.startsWith("/")
302+ ? path.slice(1)
303+ : path;
304+ const zarray = (await this.lindiFileSystemClient.readJson(
305+ pathWithoutBeginningSlash + "/.zarray",
306+ )) as ZMetaDataZArray;
307+ const zattrs = (await this.lindiFileSystemClient.readJson(
308+ pathWithoutBeginningSlash + "/.zattrs",
309+ )) as ZMetaDataZAttrs;
310+ let dataset: RemoteH5Dataset | undefined;
311+ if (zarray) {
312+ dataset = {
313+ name: getNameFromPath(path),
314+ path,
315+ shape: zarray.shape || [],
316+ dtype: zarray.dtype || "",
317+ attrs: zattrs || {},
318+ chunks: zarray.chunks,
319+ compressor: formatCompressor(zarray.compressor),
320+ filters: formatFilters(zarray.filters),
321+ };
322+ } else {
323+ dataset = undefined;
324+ }
325+ globalRemoteH5FileStats.getDatasetCount++;
326+ return dataset;
327+ }
328+ async getDatasetData(
329+ path: string,
330+ o: {
331+ slice?: [number, number][];
332+ allowBigInt?: boolean;
333+ canceler?: Canceler;
334+ },
335+ ): Promise<DatasetDataType | undefined> {
336+ // check for invalid slice
337+ if (o.slice) {
338+ for (const ss of o.slice) {
339+ if (isNaN(ss[0]) || isNaN(ss[1])) {
340+ console.warn("Invalid slice", path, o.slice);
341+ throw Error("Invalid slice");
342+ }
343+ }
344+ }
345+ if (o.slice && o.slice.length > 3) {
346+ console.warn(
347+ "Tried to slice more than three dimensions at a time",
348+ path,
349+ o.slice,
350+ );
351+ throw Error(
352+ `For now, you can't slice more than three dimensions at a time. You tried to slice ${o.slice.length} dimensions for ${path}.`,
353+ );
354+ }
355+
356+ const pathWithoutBeginningSlash = path.startsWith("/")
357+ ? path.slice(1)
358+ : path;
359+ const zarray = (await this.lindiFileSystemClient.readJson(
360+ pathWithoutBeginningSlash + "/.zarray",
361+ )) as ZMetaDataZArray | undefined;
362+ if (!zarray) {
363+ console.warn("No .zarray for", path);
364+ return undefined;
365+ }
366+
367+ // const { slice, allowBigInt, canceler } = o;
368+
369+ globalRemoteH5FileStats.getDatasetDataCount++;
370+
371+ // old system (not used by lindi)
372+ const externalHdf5 = await this.lindiFileSystemClient.readJson(
373+ pathWithoutBeginningSlash + "/.external_hdf5",
374+ );
375+ if (externalHdf5) {
376+ throw Error("External hdf5 not supported on server side");
377+ // const a = await getRemoteH5File(externalHdf5.url);
378+ // return a.getDatasetData(externalHdf5.name, o);
379+ }
380+
381+ const zattrs = (await this.lindiFileSystemClient.readJson(
382+ pathWithoutBeginningSlash + "/.zattrs",
383+ )) as ZMetaDataZAttrs;
384+ if (zattrs && zattrs["_EXTERNAL_ARRAY_LINK"]) {
385+ throw Error("External array link not supported on server side");
386+ // const externalArrayLink = zattrs["_EXTERNAL_ARRAY_LINK"];
387+ // let url0 = externalArrayLink.url;
388+ // if (this.#cacheDisabled) {
389+ // url0 += `?cacheBust=${Date.now()}`;
390+ // }
391+ // const a = await getRemoteH5File(url0);
392+ // return a.getDatasetData(externalArrayLink.name, o);
393+ }
394+
395+ const ret = await lindiDatasetDataLoader({
396+ client: this.lindiFileSystemClient,
397+ path: pathWithoutBeginningSlash,
398+ zarray,
399+ slice: o.slice || [],
400+ disableCache: this.#cacheDisabled,
401+ });
402+ if (ret.length === 1) {
403+ // candidate for scalar, need to check for _SCALAR attribute
404+ const ds = await this.getDataset(path);
405+ if (ds && ds.attrs["_SCALAR"]) {
406+ return ret[0];
407+ }
408+ }
409+ return ret;
410+ }
411+ get _lindiFileSystemClient() {
412+ return this.lindiFileSystemClient;
413+ }
414+ async getLindiZarray(path: string): Promise<ZMetaDataZArray | undefined> {
415+ const pathWithoutBeginningSlash = path.startsWith("/")
416+ ? path.slice(1)
417+ : path;
418+ return (await this.lindiFileSystemClient.readJson(
419+ pathWithoutBeginningSlash + "/.zarray",
420+ )) as ZMetaDataZArray | undefined;
421+ }
422+ getUrls() {
423+ return [this.url];
424+ }
425+ get sourceUrls(): string[] | undefined {
426+ return this.#sourceUrls;
427+ }
428+ set sourceUrls(v: string[] | undefined) {
429+ this.#sourceUrls = v;
430+ }
431+ _disableCache() {
432+ this.#cacheDisabled = true;
433+ }
434+}
435+
436+const fetchRfsFromRemoteLindi = async (
437+ url: string,
438+): Promise<{
439+ rfs: ReferenceFileSystemObject;
440+ remoteTar: RemoteTarInterface | undefined;
441+}> => {
442+ const buf: ArrayBuffer = await fetchByteRange(url, 0, 512 * 3);
443+ if (isTarHeader(buf.slice(0, 512))) {
444+ const tarEntryBuf = buf.slice(512, 512 + 1024);
445+ const tarEntryJson = new TextDecoder().decode(tarEntryBuf);
446+ const tarEntry = JSON.parse(tarEntryJson);
447+ const indexInfo = tarEntry["index"];
448+ const entryDataStartByte = indexInfo["d"];
449+ const entryDataSize = indexInfo["s"];
450+
451+ const indexBuf = await fetchByteRange(
452+ url,
453+ entryDataStartByte,
454+ entryDataSize,
455+ );
456+ const indexStr = new TextDecoder().decode(indexBuf);
457+ const index = JSON.parse(indexStr);
458+ const remoteTar: RemoteTarInterface = {
459+ url: url,
460+ getByteRangeForFile: async (fileName: string) => {
461+ const f = index.files.find((ff: any) => ff.n === fileName);
462+ if (!f) {
463+ throw Error(`File ${fileName} not found in tar`);
464+ }
465+ return {
466+ startByte: f.d as number,
467+ endByte: (f.d + f.s) as number,
468+ };
469+ },
470+ };
471+ const { startByte: rfsStartByte, endByte: rfsEndByte } =
472+ await remoteTar.getByteRangeForFile("lindi.json");
473+ const rfsBuf = await fetchByteRange(
474+ url,
475+ rfsStartByte,
476+ rfsEndByte - rfsStartByte,
477+ );
478+ const rfs = JSON.parse(new TextDecoder().decode(rfsBuf));
479+ if (!isReferenceFileSystemObject(rfs)) {
480+ console.warn(rfs);
481+ throw Error("Invalid rfs from tar");
482+ }
483+ return {
484+ rfs,
485+ remoteTar,
486+ };
487+ } else {
488+ const r = await fetch(url);
489+ if (!r.ok) throw Error("Failed to fetch LINDI file" + url);
490+ const rfs = await r.json();
491+ if (!isReferenceFileSystemObject(rfs)) {
492+ console.warn(rfs);
493+ throw Error("Invalid rfs");
494+ }
495+ return {
496+ rfs,
497+ remoteTar: undefined,
498+ };
499+ }
500+};
501+
502+const fetchByteRange = async (url: string, startByte: number, size: number) => {
503+ const r = await fetch(url, {
504+ headers: {
505+ Range: `bytes=${startByte}-${startByte + size - 1}`,
506+ },
507+ });
508+ if (!r.ok)
509+ throw Error(
510+ `Failed to fetch byte range ${startByte}-${startByte + size - 1} of ${url}`,
511+ );
512+ return await r.arrayBuffer();
513+};
514+
515+const isTarHeader = (buf: ArrayBuffer) => {
516+ if (buf.byteLength < 512) {
517+ return false;
518+ }
519+
520+ // We're only going to support ustar format
521+ // get the ustar indicator at bytes 257-262
522+ const ustarIndicator = buf.slice(257, 262);
523+ const ustarIndicatorStr = new TextDecoder().decode(
524+ ustarIndicator.slice(0, 5),
525+ );
526+ const bb = new Uint8Array(buf);
527+ if (ustarIndicatorStr === "ustar" && bb[257 + 5] == 0) {
528+ return true;
529+ }
530+
531+ // Check for any 0 bytes in the header
532+ const bb2 = new Uint8Array(buf);
533+ if (bb2.includes(0)) {
534+ console.warn(ustarIndicatorStr);
535+ throw Error(
536+ "Problem with lindi file: 0 byte found in header, but not ustar tar format",
537+ );
538+ }
539+
540+ return false;
541+};
542+
543+const getNameFromPath = (path: string) => {
544+ const parts = path.split("/");
545+ if (parts.length === 0) return "";
546+ return parts[parts.length - 1];
547+};
548+
549+const lock1: { locked: boolean } = { locked: false };
550+const globalLindiRemoteH5Files: { [url: string]: RemoteH5FileLindi } = {};
551+export const getRemoteH5FileLindi = async (url: string) => {
552+ while (lock1.locked) await new Promise((resolve) => setTimeout(resolve, 100));
553+ try {
554+ lock1.locked = true;
555+ const kk = url;
556+ if (!globalLindiRemoteH5Files[kk]) {
557+ globalLindiRemoteH5Files[kk] = await RemoteH5FileLindi.create(url);
558+ }
559+ return globalLindiRemoteH5Files[kk];
560+ } finally {
561+ lock1.locked = false;
562+ }
563+};
564+
565+// const metaOnly = (obj: ReferenceFileSystemObject) => {
566+// const ret = {
567+// refs: {} as any,
568+// version: obj.version,
569+// };
570+// for (const k in obj.refs) {
571+// if (
572+// k.endsWith(".zattrs") ||
573+// k.endsWith(".zgroup") ||
574+// k.endsWith(".zarray")
575+// ) {
576+// ret.refs[k] = obj.refs[k];
577+// }
578+// }
579+// return ret;
580+// };
581+
582+const formatCompressor = (compressor: any): string | undefined => {
583+ if (!compressor) return undefined;
584+ if (typeof compressor === "string") return compressor;
585+ if (compressor.id) return compressor.id;
586+ return JSON.stringify(compressor);
587+};
588+
589+const formatFilters = (filters: any[] | undefined): string[] | undefined => {
590+ if (!filters || filters.length === 0) return undefined;
591+ return filters.map((f) => {
592+ if (typeof f === "string") return f;
593+ if (f.id) return f.id;
594+ return JSON.stringify(f);
595+ });
596+};
597+
598+export default RemoteH5FileLindi;
src/remote-h5-file/lib/lindi/decodeMp4.tsadded+98−0View file
@@ -0,0 +1,98 @@
1+// This was a nice attempt but in the end
2+// it took way too long to decode a chunk
3+
4+export const decodeMp4 = async (
5+ chunk: ArrayBuffer,
6+ numFrames: number,
7+ width: number,
8+ height: number,
9+) => {
10+ const blob = new Blob([chunk], { type: "video/mp4" });
11+ const url = URL.createObjectURL(blob);
12+
13+ const numBatches = 10;
14+ const framesPerBatch = Math.ceil(numFrames / numBatches);
15+ const batches = [];
16+ for (let i = 0; i < numBatches; i++) {
17+ const start = i * framesPerBatch;
18+ const end = Math.min(start + framesPerBatch, numFrames);
19+ batches.push({
20+ start,
21+ end,
22+ url,
23+ width,
24+ height,
25+ });
26+ }
27+
28+ // do batches in parallel
29+ const promises = batches.map((batch) => doBatch(batch));
30+ const results = await Promise.all(promises);
31+ const frames = results.flat();
32+
33+ // Cleanup
34+ URL.revokeObjectURL(url);
35+
36+ return frames;
37+};
38+
39+const doBatch = async (batch: {
40+ start: number;
41+ end: number;
42+ url: string;
43+ width: number;
44+ height: number;
45+}) => {
46+ const { start, end, url, width, height } = batch;
47+ // Prepare video element
48+ const video = document.createElement("video");
49+ video.width = width;
50+ video.height = height;
51+ video.src = url;
52+ video.muted = true;
53+ const canvas = document.createElement("canvas");
54+ canvas.width = width;
55+ canvas.height = height;
56+ const ctx = canvas.getContext("2d");
57+ if (!ctx) throw Error("Unable to create canvas context");
58+
59+ video.onerror = (e) => {
60+ console.log("Video error", e, video.error);
61+ };
62+
63+ // Wait for video to load
64+ await new Promise((resolve) => {
65+ video.addEventListener("loadedmetadata", resolve, { once: true });
66+ });
67+
68+ // Utility to seek video and extract frame
69+ const extractFrame = (time: number) =>
70+ new Promise((resolve) => {
71+ video.currentTime = time;
72+ video.addEventListener(
73+ "seeked",
74+ () => {
75+ ctx.drawImage(video, 0, 0, canvas.width, canvas.height);
76+ const imageData = ctx.getImageData(0, 0, canvas.width, canvas.height);
77+ resolve(imageData.data);
78+ },
79+ { once: true },
80+ );
81+ });
82+
83+ // Step 4 & 5: Extract frames (simplified)
84+ const frames = [];
85+ for (let i = start; i < end; i++) {
86+ const time = i; // fps = 1
87+ // print every 10 frames
88+ if (frames.length % 10 === 0) {
89+ console.log("Frames", frames.length);
90+ }
91+ const frameData = await extractFrame(time);
92+ frames.push(frameData); // This pushes 3D data; needs adjustment for 4D structure
93+ }
94+
95+ console.log("Frames", frames);
96+
97+ return frames; // This is your 4D array
98+};
src/remote-h5-file/lib/lindi/fft.tsadded+234−0View file
@@ -0,0 +1,234 @@
1+// jfm changed Float32Array to Float32Array
2+
3+/* eslint-disable @typescript-eslint/no-inferrable-types */
4+/* eslint-disable @typescript-eslint/no-unused-vars */
5+/* eslint-disable prefer-const */
6+/*
7+ * Free FFT and convolution (TypeScript)
8+ *
9+ * Copyright (c) 2022 Project Nayuki. (MIT License)
10+ * https://www.nayuki.io/page/free-small-fft-in-multiple-languages
11+ *
12+ * Permission is hereby granted, free of charge, to any person obtaining a copy of
13+ * this software and associated documentation files (the "Software"), to deal in
14+ * the Software without restriction, including without limitation the rights to
15+ * use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
16+ * the Software, and to permit persons to whom the Software is furnished to do so,
17+ * subject to the following conditions:
18+ * - The above copyright notice and this permission notice shall be included in
19+ * all copies or substantial portions of the Software.
20+ * - The Software is provided "as is", without warranty of any kind, express or
21+ * implied, including but not limited to the warranties of merchantability,
22+ * fitness for a particular purpose and noninfringement. In no event shall the
23+ * authors or copyright holders be liable for any claim, damages or other
24+ * liability, whether in an action of contract, tort or otherwise, arising from,
25+ * out of or in connection with the Software or the use or other dealings in the
26+ * Software.
27+ */
28+
29+/*
30+ * Computes the discrete Fourier transform (DFT) of the given complex vector, storing the result back into the vector.
31+ * The vector can have any length. This is a wrapper function.
32+ */
33+export function transform(
34+ real: Array<number> | Float32Array,
35+ imag: Array<number> | Float32Array,
36+): void {
37+ const n: number = real.length;
38+ if (n != imag.length) throw new RangeError("Mismatched lengths");
39+ if (n == 0) return;
40+ else if ((n & (n - 1)) == 0)
41+ // Is power of 2
42+ transformRadix2(real, imag); // More complicated algorithm for arbitrary sizes
43+ else transformBluestein(real, imag);
44+}
45+
46+/*
47+ * Computes the inverse discrete Fourier transform (IDFT) of the given complex vector, storing the result back into the vector.
48+ * The vector can have any length. This is a wrapper function. This transform does not perform scaling, so the inverse is not a true inverse.
49+ */
50+export function inverseTransform(
51+ real: Array<number> | Float32Array,
52+ imag: Array<number> | Float32Array,
53+): void {
54+ transform(imag, real);
55+}
56+
57+/*
58+ * Computes the discrete Fourier transform (DFT) of the given complex vector, storing the result back into the vector.
59+ * The vector's length must be a power of 2. Uses the Cooley-Tukey decimation-in-time radix-2 algorithm.
60+ */
61+function transformRadix2(
62+ real: Array<number> | Float32Array,
63+ imag: Array<number> | Float32Array,
64+): void {
65+ // Length variables
66+ const n: number = real.length;
67+ if (n != imag.length) throw new RangeError("Mismatched lengths");
68+ if (n == 1)
69+ // Trivial transform
70+ return;
71+ let levels: number = -1;
72+ for (let i = 0; i < 32; i++) {
73+ if (1 << i == n) levels = i; // Equal to log2(n)
74+ }
75+ if (levels == -1) throw new RangeError("Length is not a power of 2");
76+
77+ // Trigonometric tables
78+ let cosTable = new Array<number>(n / 2);
79+ let sinTable = new Array<number>(n / 2);
80+ for (let i = 0; i < n / 2; i++) {
81+ cosTable[i] = Math.cos((2 * Math.PI * i) / n);
82+ sinTable[i] = Math.sin((2 * Math.PI * i) / n);
83+ }
84+
85+ // Bit-reversed addressing permutation
86+ for (let i = 0; i < n; i++) {
87+ const j: number = reverseBits(i, levels);
88+ if (j > i) {
89+ let temp: number = real[i];
90+ real[i] = real[j];
91+ real[j] = temp;
92+ temp = imag[i];
93+ imag[i] = imag[j];
94+ imag[j] = temp;
95+ }
96+ }
97+
98+ // Cooley-Tukey decimation-in-time radix-2 FFT
99+ for (let size = 2; size <= n; size *= 2) {
100+ const halfsize: number = size / 2;
101+ const tablestep: number = n / size;
102+ for (let i = 0; i < n; i += size) {
103+ for (let j = i, k = 0; j < i + halfsize; j++, k += tablestep) {
104+ const l: number = j + halfsize;
105+ const tpre: number = real[l] * cosTable[k] + imag[l] * sinTable[k];
106+ const tpim: number = -real[l] * sinTable[k] + imag[l] * cosTable[k];
107+ real[l] = real[j] - tpre;
108+ imag[l] = imag[j] - tpim;
109+ real[j] += tpre;
110+ imag[j] += tpim;
111+ }
112+ }
113+ }
114+
115+ // Returns the integer whose value is the reverse of the lowest 'width' bits of the integer 'val'.
116+ function reverseBits(val: number, width: number): number {
117+ let result: number = 0;
118+ for (let i = 0; i < width; i++) {
119+ result = (result << 1) | (val & 1);
120+ val >>>= 1;
121+ }
122+ return result;
123+ }
124+}
125+
126+/*
127+ * Computes the discrete Fourier transform (DFT) of the given complex vector, storing the result back into the vector.
128+ * The vector can have any length. This requires the convolution function, which in turn requires the radix-2 FFT function.
129+ * Uses Bluestein's chirp z-transform algorithm.
130+ */
131+function transformBluestein(
132+ real: Array<number> | Float32Array,
133+ imag: Array<number> | Float32Array,
134+): void {
135+ // Find a power-of-2 convolution length m such that m >= n * 2 + 1
136+ const n: number = real.length;
137+ if (n != imag.length) throw new RangeError("Mismatched lengths");
138+ let m: number = 1;
139+ while (m < n * 2 + 1) m *= 2;
140+
141+ // Trigonometric tables
142+ let cosTable = new Array<number>(n);
143+ let sinTable = new Array<number>(n);
144+ for (let i = 0; i < n; i++) {
145+ const j: number = (i * i) % (n * 2); // This is more accurate than j = i * i
146+ cosTable[i] = Math.cos((Math.PI * j) / n);
147+ sinTable[i] = Math.sin((Math.PI * j) / n);
148+ }
149+
150+ // Temporary vectors and preprocessing
151+ let areal: Array<number> = newArrayOfZeros(m);
152+ let aimag: Array<number> = newArrayOfZeros(m);
153+ for (let i = 0; i < n; i++) {
154+ areal[i] = real[i] * cosTable[i] + imag[i] * sinTable[i];
155+ aimag[i] = -real[i] * sinTable[i] + imag[i] * cosTable[i];
156+ }
157+ let breal: Array<number> = newArrayOfZeros(m);
158+ let bimag: Array<number> = newArrayOfZeros(m);
159+ breal[0] = cosTable[0];
160+ bimag[0] = sinTable[0];
161+ for (let i = 1; i < n; i++) {
162+ breal[i] = breal[m - i] = cosTable[i];
163+ bimag[i] = bimag[m - i] = sinTable[i];
164+ }
165+
166+ // Convolution
167+ let creal = new Array<number>(m);
168+ let cimag = new Array<number>(m);
169+ convolveComplex(areal, aimag, breal, bimag, creal, cimag);
170+
171+ // Postprocessing
172+ for (let i = 0; i < n; i++) {
173+ real[i] = creal[i] * cosTable[i] + cimag[i] * sinTable[i];
174+ imag[i] = -creal[i] * sinTable[i] + cimag[i] * cosTable[i];
175+ }
176+}
177+
178+/*
179+ * Computes the circular convolution of the given real vectors. Each vector's length must be the same.
180+ */
181+// function convolveReal(xvec: Array<number>|Float32Array, yvec: Array<number>|Float32Array, outvec: Array<number>|Float32Array): void {
182+// const n: number = xvec.length;
183+// if (n != yvec.length || n != outvec.length)
184+// throw new RangeError("Mismatched lengths");
185+// convolveComplex(xvec, newArrayOfZeros(n), yvec, newArrayOfZeros(n), outvec, newArrayOfZeros(n));
186+// }
187+
188+/*
189+ * Computes the circular convolution of the given complex vectors. Each vector's length must be the same.
190+ */
191+function convolveComplex(
192+ xreal: Array<number> | Float32Array,
193+ ximag: Array<number> | Float32Array,
194+ yreal: Array<number> | Float32Array,
195+ yimag: Array<number> | Float32Array,
196+ outreal: Array<number> | Float32Array,
197+ outimag: Array<number> | Float32Array,
198+): void {
199+ const n: number = xreal.length;
200+ if (
201+ n != ximag.length ||
202+ n != yreal.length ||
203+ n != yimag.length ||
204+ n != outreal.length ||
205+ n != outimag.length
206+ )
207+ throw new RangeError("Mismatched lengths");
208+
209+ xreal = xreal.slice();
210+ ximag = ximag.slice();
211+ yreal = yreal.slice();
212+ yimag = yimag.slice();
213+ transform(xreal, ximag);
214+ transform(yreal, yimag);
215+
216+ for (let i = 0; i < n; i++) {
217+ const temp: number = xreal[i] * yreal[i] - ximag[i] * yimag[i];
218+ ximag[i] = ximag[i] * yreal[i] + xreal[i] * yimag[i];
219+ xreal[i] = temp;
220+ }
221+ inverseTransform(xreal, ximag);
222+
223+ for (let i = 0; i < n; i++) {
224+ // Scaling (because this FFT implementation omits it)
225+ outreal[i] = xreal[i] / n;
226+ outimag[i] = ximag[i] / n;
227+ }
228+}
229+
230+function newArrayOfZeros(n: number): Array<number> {
231+ let result: Array<number> = [];
232+ for (let i = 0; i < n; i++) result.push(0);
233+ return result;
234+}
src/remote-h5-file/lib/lindi/lindiDatasetDataLoader.tsadded+500−0View file
@@ -0,0 +1,500 @@
1+/* eslint-disable @typescript-eslint/no-explicit-any */
2+import ReferenceFileSystemClient from "./ReferenceFileSystemClient";
3+import { ZarrFileSystemClient, ZMetaDataZArray } from "./RemoteH5FileLindi";
4+
5+const lindiDatasetDataLoader = async (o: {
6+ client: ReferenceFileSystemClient | ZarrFileSystemClient;
7+ path: string;
8+ zarray: ZMetaDataZArray;
9+ slice: [number, number][];
10+ assertSingleChunkInFirstTwoDimensions?: boolean;
11+ disableCache?: boolean;
12+}) => {
13+ const { client, zarray, path, slice, assertSingleChunkInFirstTwoDimensions } =
14+ o;
15+
16+ const chunkShape = zarray.chunks;
17+ const shape = zarray.shape;
18+ const dtype = zarray.dtype;
19+ if (!chunkShape) throw Error("No chunks shape for " + path);
20+ if (!shape) throw Error("No shape for " + path);
21+ if (!dtype) throw Error("No dtype for " + path);
22+ const ndims = shape.length;
23+ if (ndims !== chunkShape.length)
24+ throw Error("Mismatched ndims and chunk shape for " + path);
25+
26+ if (o.slice.length === 3) {
27+ // in this case we slice by two and then return the result of slicing by the third
28+ const slice1 = slice.slice(0, 2);
29+ const sN1 = slice1[0][1] - slice1[0][0];
30+ const sN2 = slice1[1][1] - slice1[1][0];
31+ const sN3 = o.slice[2][1] - o.slice[2][0];
32+ const sNother = shape.slice(3).reduce((a, b) => a * b, 1);
33+ const N3 = shape[2];
34+ const xx = await lindiDatasetDataLoader({
35+ client,
36+ path,
37+ zarray,
38+ slice: slice1,
39+ assertSingleChunkInFirstTwoDimensions,
40+ disableCache: o.disableCache,
41+ });
42+ const xxRet = allocateArrayWithDtype(sN1 * sN2 * sN3 * sNother, dtype);
43+ let iRet = 0;
44+ for (let i1 = 0; i1 < sN1; i1++) {
45+ for (let i2 = 0; i2 < sN2; i2++) {
46+ for (let i3 = o.slice[2][0]; i3 < o.slice[2][1]; i3++) {
47+ for (let i4 = 0; i4 < sNother; i4++) {
48+ xxRet[iRet] = xx[i4 + sNother * (i3 + N3 * (i2 + sN2 * i1))];
49+ iRet++;
50+ }
51+ }
52+ }
53+ }
54+ return xxRet;
55+ }
56+ if (o.slice.length > 3) {
57+ throw Error(
58+ `For now, you can't slice more than three dimensions at a time. You tried to slice ${o.slice.length} dimensions for ${path}.`,
59+ );
60+ }
61+
62+ const macroChunkShape = chunkShape.map((cs, i) => Math.ceil(shape[i] / cs));
63+
64+ // check if we have a single chunk with no filters or compression (single contiguous block of data)
65+ // It's important to handle this case specially because in this situation we don't need to download
66+ // the entire chunk, we can just download the slice we need.
67+ const singleChunk = macroChunkShape.reduce((a, b) => a * b, 1) === 1;
68+ const noFiltersOrCompression =
69+ !zarray.compressor && (!zarray.filters || zarray.filters.length === 0);
70+ if (singleChunk && noFiltersOrCompression && slice && slice.length > 0) {
71+ if (slice.length > 2) {
72+ throw Error(
73+ "For now, you can only slice two dimensions at a time for single chunk contiguous data",
74+ );
75+ }
76+ const dtypeByteSize = getDtypeByteSize(dtype);
77+ const startByte =
78+ slice[0][0] * shape.slice(1).reduce((a, b) => a * b, 1) * dtypeByteSize;
79+ const endByte =
80+ slice[0][1] * shape.slice(1).reduce((a, b) => a * b, 1) * dtypeByteSize;
81+ let singleChunkPath = path + "/0";
82+ for (let i = 1; i < ndims; i++) {
83+ singleChunkPath += ".0";
84+ }
85+ const dd = await client.readBinary(singleChunkPath, {
86+ decodeArray: false,
87+ startByte,
88+ endByte,
89+ });
90+ let a = createDataView(dd, dtype);
91+ if (slice.length === 2) {
92+ if (shape.length === 1) {
93+ if (slice[1][0] !== 0 || slice[1][1] !== 1) {
94+ throw Error(
95+ `For now, you can't slice the second dimension for single chunk contiguous data`,
96+ );
97+ }
98+ return a;
99+ }
100+ const ss = shape.slice(2).reduce((a, b) => a * b, 1);
101+ const newRet = allocateArrayWithDtype(
102+ (slice[0][1] - slice[0][0]) * (slice[1][1] - slice[1][0]) * ss,
103+ dtype,
104+ );
105+ let iRet = 0;
106+ for (let i = 0; i < slice[0][1] - slice[0][0]; i++) {
107+ for (let j = slice[1][0]; j < slice[1][1]; j++) {
108+ for (let k = 0; k < ss; k++) {
109+ newRet[iRet] = a[(i * shape[1] + j) * ss + k];
110+ iRet++;
111+ }
112+ }
113+ }
114+ a = newRet as any;
115+ }
116+ return a;
117+ }
118+
119+ const prodChunkSizeOfAllButFirstDimension = chunkShape
120+ .slice(1)
121+ .reduce((a, b) => a * b, 1);
122+ const prodChunkSizeOfAllButFirstTwoDimensions = chunkShape
123+ .slice(2)
124+ .reduce((a, b) => a * b, 1);
125+ const prodShapeSizeOfAllButFirstTwoDimensions = shape
126+ .slice(2)
127+ .reduce((a, b) => a * b, 1);
128+ const prodMacroChunkShapeAllButFirstTwoDimensions = macroChunkShape
129+ .slice(2)
130+ .reduce((a, b) => a * b, 1);
131+
132+ let i1Start = 0;
133+ let i1End = shape[0];
134+ let i2Start = 0;
135+ let i2End = ndims > 1 ? shape[1] : 1;
136+ if (slice) {
137+ if (slice.length >= 1) {
138+ i1Start = slice[0][0];
139+ i1End = slice[0][1];
140+ }
141+ if (slice.length >= 2) {
142+ i2Start = slice[1][0];
143+ i2End = slice[1][1];
144+ }
145+ if (slice.length > 2) {
146+ throw Error(
147+ `For now, you can't slice more than two dimensions at a time. You tried to slice ${slice.length} dimensions for ${path}.`,
148+ );
149+ }
150+ }
151+ if (i1End > shape[0]) i1End = shape[0];
152+ if (i2End > shape[1]) i2End = shape[1];
153+
154+ const shape2 = ndims > 1 ? shape[1] : 1;
155+ const chunkShape2 = ndims > 1 ? chunkShape[1] : 1;
156+
157+ if (i1Start < 0) {
158+ throw Error(`Problem slicing ${path}: i1Start < 0: ${i1Start}`);
159+ }
160+ if (i1End > shape[0]) {
161+ throw Error(
162+ `Problem slicing ${path}: i1End > shape[0]: ${i1End} > ${shape[0]}`,
163+ );
164+ }
165+ if (i2Start < 0) {
166+ throw Error(`Problem slicing ${path}: i2Start < 0: ${i2Start}`);
167+ }
168+ if (i2End > shape2) {
169+ throw Error(
170+ `Problem slicing ${path}: i2End > shape[1]: ${i2End} > ${chunkShape2}`,
171+ );
172+ }
173+
174+ const i1StartChunk = Math.floor(i1Start / chunkShape[0]);
175+ const i1EndChunk = Math.floor((i1End - 1) / chunkShape[0]);
176+ const i2StartChunk = ndims > 1 ? Math.floor(i2Start / chunkShape[1]) : 0;
177+ const i2EndChunk = ndims > 1 ? Math.floor((i2End - 1) / chunkShape[1]) : 0;
178+ if (i1StartChunk === i1EndChunk && i2StartChunk === i2EndChunk) {
179+ // With respect to the first two dimensions,
180+ // we are entirely within a single chunk.
181+
182+ if (prodMacroChunkShapeAllButFirstTwoDimensions === 1) {
183+ // in this case we are truly in a single chunk because there is only one chunk in the other dimensions
184+ let chunkPath = path + "/" + i1StartChunk;
185+ if (ndims > 1) {
186+ chunkPath += "." + i2StartChunk;
187+ }
188+ for (let d = 2; d < ndims; d++) {
189+ chunkPath += ".0";
190+ }
191+ const x = await client.readBinary(chunkPath, {
192+ decodeArray: true,
193+ disableCache: o.disableCache,
194+ });
195+ if (!x) {
196+ console.log({
197+ i1StartChunk,
198+ i1EndChunk,
199+ i2StartChunk,
200+ i2EndChunk,
201+ i1Start,
202+ i1End,
203+ i2Start,
204+ i2End,
205+ shape,
206+ chunkShape,
207+ });
208+ throw Error("Unable to read chunk: " + chunkPath);
209+ }
210+ const j1Start = i1Start - i1StartChunk * chunkShape[0];
211+ const j1End = i1End - i1StartChunk * chunkShape[0];
212+ const j2Start = i2Start - i2StartChunk * chunkShape2;
213+ const j2End = i2End - i2StartChunk * chunkShape2;
214+ const slicingInSecondDimension =
215+ ndims > 1 && (j2Start > 0 || j2End < chunkShape2);
216+ if (!slicingInSecondDimension) {
217+ // we are not slicing in second dimension. In this case we don't need to make a copy of the data
218+ const ret = x.slice(
219+ j1Start * prodChunkSizeOfAllButFirstDimension,
220+ j1End * prodChunkSizeOfAllButFirstDimension,
221+ );
222+ return ret;
223+ } else {
224+ // we are slicing in second dimension, so we need to make a copy of the data
225+ const ret = allocateArrayWithDtype(
226+ (i1End - i1Start) *
227+ (i2End - i2Start) *
228+ prodShapeSizeOfAllButFirstTwoDimensions,
229+ dtype,
230+ );
231+ let iRet = 0;
232+ for (let j1 = j1Start; j1 < j1End; j1++) {
233+ for (let j2 = j2Start; j2 < j2End; j2++) {
234+ for (
235+ let j3 = 0;
236+ j3 < prodShapeSizeOfAllButFirstTwoDimensions;
237+ j3++
238+ ) {
239+ ret[iRet] =
240+ x[
241+ (j1 * chunkShape2 + j2) *
242+ prodChunkSizeOfAllButFirstTwoDimensions +
243+ j3
244+ ];
245+ iRet++;
246+ }
247+ }
248+ }
249+ return ret;
250+ }
251+ } else {
252+ // there is more than one chunk in the other dimensions, and we need to concatenate them
253+ if (ndims > 4) {
254+ throw Error("Case not yet supported: C2");
255+ }
256+ while (macroChunkShape.length < 4) {
257+ macroChunkShape.push(1);
258+ }
259+ const retList = [];
260+ for (let iii3 = 0; iii3 < macroChunkShape[2]; iii3++) {
261+ for (let iii4 = 0; iii4 < macroChunkShape[3]; iii4++) {
262+ let chunkPath = path + "/" + i1StartChunk;
263+ chunkPath += "." + i2StartChunk;
264+ chunkPath += "." + iii3;
265+ if (ndims === 4) {
266+ chunkPath += "." + iii4;
267+ }
268+ const x = await client.readBinary(chunkPath, {
269+ decodeArray: true,
270+ disableCache: o.disableCache,
271+ });
272+ if (!x) {
273+ console.log({
274+ i1StartChunk,
275+ i1EndChunk,
276+ i2StartChunk,
277+ i2EndChunk,
278+ i1Start,
279+ i1End,
280+ i2Start,
281+ i2End,
282+ shape,
283+ chunkShape,
284+ });
285+ throw Error("Unable to read chunk: " + chunkPath);
286+ }
287+ const j1Start = i1Start - i1StartChunk * chunkShape[0];
288+ const j1End = i1End - i1StartChunk * chunkShape[0];
289+ const j2Start = i2Start - i2StartChunk * chunkShape2;
290+ const j2End = i2End - i2StartChunk * chunkShape2;
291+ const slicingInSecondDimension =
292+ ndims > 1 && (j2Start > 0 || j2End < chunkShape2);
293+ if (!slicingInSecondDimension) {
294+ // we are not slicing in second dimension. In this case we don't need to make a copy of the data
295+ const ret0 = x.slice(
296+ j1Start * prodChunkSizeOfAllButFirstDimension,
297+ j1End * prodChunkSizeOfAllButFirstDimension,
298+ );
299+ retList.push(ret0);
300+ } else {
301+ // we are slicing in second dimension, so we need to make a copy of the data
302+ const ret0 = allocateArrayWithDtype(
303+ (i1End - i1Start) *
304+ (i2End - i2Start) *
305+ prodChunkSizeOfAllButFirstTwoDimensions,
306+ dtype,
307+ );
308+ let iRet0 = 0;
309+ for (let j1 = j1Start; j1 < j1End; j1++) {
310+ for (let j2 = j2Start; j2 < j2End; j2++) {
311+ for (
312+ let j3 = 0;
313+ j3 < prodChunkSizeOfAllButFirstTwoDimensions;
314+ j3++
315+ ) {
316+ ret0[iRet0] =
317+ x[
318+ (j1 * chunkShape2 + j2) *
319+ prodChunkSizeOfAllButFirstTwoDimensions +
320+ j3
321+ ];
322+ iRet0++;
323+ }
324+ }
325+ }
326+ retList.push(ret0);
327+ }
328+ }
329+ }
330+ // now concatenate the ret0s
331+ const ret = allocateArrayWithDtype(
332+ (i1End - i1Start) *
333+ (i2End - i2Start) *
334+ prodShapeSizeOfAllButFirstTwoDimensions,
335+ dtype,
336+ );
337+ let iRet = 0;
338+ for (let i1 = 0; i1 < i1End - i1Start; i1++) {
339+ for (let i2 = 0; i2 < i2End - i2Start; i2++) {
340+ for (let i = 0; i < retList.length; i++) {
341+ for (
342+ let i3 = 0;
343+ i3 < prodChunkSizeOfAllButFirstTwoDimensions;
344+ i3++
345+ ) {
346+ ret[iRet] =
347+ retList[i][
348+ i1 *
349+ (i2End - i2Start) *
350+ prodChunkSizeOfAllButFirstTwoDimensions +
351+ i2 * prodChunkSizeOfAllButFirstTwoDimensions +
352+ i3
353+ ];
354+ iRet++;
355+ }
356+ }
357+ }
358+ }
359+ return ret;
360+ }
361+ }
362+
363+ if (assertSingleChunkInFirstTwoDimensions)
364+ throw Error(
365+ "Unexpected case. We should have handled all cases by now (assertSingleChunkInFirstTwoDimensions)",
366+ );
367+
368+ const ret = allocateArrayWithDtype(
369+ (i1End - i1Start) *
370+ (i2End - i2Start) *
371+ prodShapeSizeOfAllButFirstTwoDimensions,
372+ dtype,
373+ );
374+
375+ const handleChunk = async (o2: {
376+ slice1: [number, number];
377+ slice2: [number, number];
378+ }) => {
379+ const { slice1, slice2 } = o2;
380+ const sliceA = [slice1, slice2];
381+ const xx = await lindiDatasetDataLoader({
382+ client,
383+ path,
384+ zarray,
385+ slice: sliceA,
386+ assertSingleChunkInFirstTwoDimensions: true, // avoid infinite recursion by accident
387+ disableCache: o.disableCache,
388+ });
389+ let iXX = 0;
390+ for (let ii1 = slice1[0]; ii1 < slice1[1]; ii1++) {
391+ for (let ii2 = slice2[0]; ii2 < slice2[1]; ii2++) {
392+ let iRet =
393+ (ii1 - i1Start) *
394+ (i2End - i2Start) *
395+ prodShapeSizeOfAllButFirstTwoDimensions +
396+ (ii2 - i2Start) * prodShapeSizeOfAllButFirstTwoDimensions;
397+ for (
398+ let ii3 = 0;
399+ ii3 < prodShapeSizeOfAllButFirstTwoDimensions;
400+ ii3++
401+ ) {
402+ ret[iRet] = xx[iXX];
403+ iRet++;
404+ iXX++;
405+ }
406+ }
407+ }
408+ };
409+
410+ const promises: Promise<void>[] = [];
411+ for (let i1Chunk = i1StartChunk; i1Chunk <= i1EndChunk; i1Chunk++) {
412+ let slice1: [number, number];
413+ if (i1Chunk === i1StartChunk && i1Chunk === i1EndChunk) {
414+ slice1 = [i1Start, i1End];
415+ } else if (i1Chunk === i1StartChunk) {
416+ slice1 = [i1Start, (i1Chunk + 1) * chunkShape[0]];
417+ } else if (i1Chunk === i1EndChunk) {
418+ slice1 = [i1Chunk * chunkShape[0], i1End];
419+ } else {
420+ slice1 = [i1Chunk * chunkShape[0], (i1Chunk + 1) * chunkShape[0]];
421+ }
422+ for (let i2Chunk = i2StartChunk; i2Chunk <= i2EndChunk; i2Chunk++) {
423+ let slice2: [number, number];
424+ if (i2Chunk === i2StartChunk && i2Chunk === i2EndChunk) {
425+ slice2 = [i2Start, i2End];
426+ } else if (i2Chunk === i2StartChunk) {
427+ slice2 = [i2Start, (i2Chunk + 1) * chunkShape2];
428+ } else if (i2Chunk === i2EndChunk) {
429+ slice2 = [i2Chunk * chunkShape2, i2End];
430+ } else {
431+ slice2 = [i2Chunk * chunkShape2, (i2Chunk + 1) * chunkShape2];
432+ }
433+
434+ promises.push(handleChunk({ slice1, slice2 }));
435+ }
436+ }
437+ await Promise.all(promises);
438+ return ret;
439+};
440+
441+const allocateArrayWithDtype = (size: number, dtype: string) => {
442+ if (dtype === "<f4") return new Float32Array(size);
443+ if (dtype === "<f8") return new Float64Array(size);
444+ if (dtype === "<i1" || dtype === "|i1") return new Int8Array(size);
445+ if (dtype === "<i2") return new Int16Array(size);
446+ if (dtype === "<i4") return new Int32Array(size);
447+ if (dtype === "<i8") {
448+ const a = new BigInt64Array(size);
449+ // convert to regular Int32Array because js has trouble mixing BigInt64Array with other numbers
450+ const ret = new Int32Array(size);
451+ for (let i = 0; i < size; i++) {
452+ ret[i] = Number(a[i]);
453+ }
454+ return ret;
455+ }
456+ if (dtype === "<u1" || dtype === "|u1") return new Uint8Array(size);
457+ if (dtype === "<u2") return new Uint16Array(size);
458+ if (dtype === "<u4") return new Uint32Array(size);
459+ if (dtype === "<u8") {
460+ const a = new BigUint64Array(size);
461+ // convert to regular Uint32Array because js has trouble mixing BigUint64Array with other numbers
462+ const ret = new Uint32Array(size);
463+ for (let i = 0; i < size; i++) {
464+ ret[i] = Number(a[i]);
465+ }
466+ return ret;
467+ }
468+ if (dtype === "|O") return new Array(size);
469+ throw Error(`Unsupported dtype: ${dtype}`);
470+};
471+
472+const createDataView = (dd: ArrayBuffer, dtype: string) => {
473+ if (dtype === "<f4") return new Float32Array(dd);
474+ if (dtype === "<f8") return new Float64Array(dd);
475+ if (dtype === "<i1" || dtype === "|i1") return new Int8Array(dd);
476+ if (dtype === "<i2") return new Int16Array(dd);
477+ if (dtype === "<i4") return new Int32Array(dd);
478+ if (dtype === "<i8") return new BigInt64Array(dd);
479+ if (dtype === "<u1" || dtype === "|u1") return new Uint8Array(dd);
480+ if (dtype === "<u2") return new Uint16Array(dd);
481+ if (dtype === "<u4") return new Uint32Array(dd);
482+ if (dtype === "<u8") return new BigUint64Array(dd);
483+ throw Error(`Unsupported dtype: ${dtype}`);
484+};
485+
486+const getDtypeByteSize = (dtype: string) => {
487+ if (dtype === "<f4") return 4;
488+ if (dtype === "<f8") return 8;
489+ if (dtype === "<i1" || dtype === "|i1") return 1;
490+ if (dtype === "<i2") return 2;
491+ if (dtype === "<i4") return 4;
492+ if (dtype === "<i8") return 8;
493+ if (dtype === "<u1" || dtype === "|u1") return 1;
494+ if (dtype === "<u2") return 2;
495+ if (dtype === "<u4") return 4;
496+ if (dtype === "<u8") return 8;
497+ throw Error(`Unsupported dtype: ${dtype}`);
498+};
499+
500+export default lindiDatasetDataLoader;
src/remote-h5-file/lib/lindi/qfc.tsadded+344−0View file
@@ -0,0 +1,344 @@
1+// @ts-nocheck
2+import pako from "pako";
3+import { inverseTransform } from "./fft";
4+
5+type QfcCompressionOpts = {
6+ compression_method: "zlib" | "zstd";
7+ dtype: "float32" | "int16";
8+ id: "qfc";
9+ quant_scale_factor: number;
10+ segment_length: number;
11+ zlib_level: number;
12+ zstd_level: number;
13+};
14+
15+const isQfcCompressionOpts = (x: any): x is QfcCompressionOpts => {
16+ if (!x) return false;
17+ if (typeof x !== "object") return false;
18+ if (x.compression_method !== "zlib" && x.compression_method !== "zstd")
19+ return false;
20+ if (x.dtype !== "float32" && x.dtype !== "int16") return false;
21+ if (x.id !== "qfc") return false;
22+ if (typeof x.quant_scale_factor !== "number") return false;
23+ if (typeof x.segment_length !== "number") return false;
24+ if (typeof x.zlib_level !== "number") return false;
25+ if (typeof x.zstd_level !== "number") return false;
26+ return true;
27+};
28+
29+export const qfcDecompress = async (
30+ buf: ArrayBuffer,
31+ shape: number[],
32+ compressor: QfcCompressionOpts,
33+): Promise<any> => {
34+ if (!isQfcCompressionOpts(compressor)) {
35+ console.warn(compressor);
36+ throw Error("Invalid qfc compressor");
37+ }
38+
39+ const header = new Int32Array(buf, 0, 5);
40+ if (header[0] !== 7364182) {
41+ throw Error(`Invalid header[0]: ${header[0]}`);
42+ }
43+ if (header[1] !== 1) {
44+ throw Error(`Invalid header[1]: ${header[1]}`);
45+ }
46+ const num_samples = header[2];
47+ const num_channels = header[3];
48+ if (num_samples !== shape[0]) {
49+ throw Error(
50+ `Unexpected num samples in header. Expected ${shape[0]}, got ${num_samples}`,
51+ );
52+ }
53+ if (num_channels !== shape[1]) {
54+ throw Error(
55+ `Unexpected num channels in header. Expected ${shape[1]}, got ${num_channels}`,
56+ );
57+ }
58+ if (header[4] !== compressor.segment_length) {
59+ throw Error(
60+ `Unexpected segment length in header. Expected ${compressor.segment_length}, got ${header[4]}`,
61+ );
62+ }
63+
64+ const decompressed_buf = await qfc_multi_segment_decompress({
65+ buf: buf.slice(4 * 5),
66+ dtype: compressor.dtype,
67+ num_channels,
68+ num_samples,
69+ segment_length: compressor.segment_length,
70+ quant_scale_factor: compressor.quant_scale_factor,
71+ compression_method: compressor.compression_method,
72+ });
73+
74+ if (
75+ decompressed_buf.byteLength !==
76+ num_samples * num_channels * (compressor.dtype === "float32" ? 4 : 2)
77+ ) {
78+ console.warn("compressor", compressor);
79+ throw Error(
80+ `Unexpected decompressed buffer length. Expected ${num_samples * num_channels * (compressor.dtype === "float32" ? 4 : 2)}, got ${decompressed_buf.byteLength}`,
81+ );
82+ }
83+
84+ return decompressed_buf;
85+};
86+
87+const qfc_multi_segment_decompress = async (o: {
88+ buf: ArrayBuffer;
89+ dtype: "float32" | "int16";
90+ num_channels: number;
91+ num_samples: number;
92+ segment_length: number;
93+ quant_scale_factor: number;
94+ compression_method: "zlib" | "zstd";
95+}): Promise<ArrayBuffer> => {
96+ const {
97+ buf,
98+ dtype,
99+ num_channels,
100+ num_samples,
101+ segment_length,
102+ quant_scale_factor,
103+ compression_method,
104+ } = o;
105+
106+ let decompressedArray: Int16Array;
107+ if (compression_method === "zlib") {
108+ decompressedArray = new Int16Array(pako.inflate(buf).buffer);
109+ } else if (compression_method === "zstd") {
110+ throw Error("zstd decompression not implemented");
111+ } else {
112+ throw Error(`Unexpected compression method: ${compression_method}`);
113+ }
114+
115+ return await qfc_multi_segment_inv_pre_compress({
116+ array: decompressedArray,
117+ quant_scale_factor,
118+ segment_length,
119+ dtype,
120+ num_channels,
121+ num_samples,
122+ });
123+};
124+
125+const qfc_multi_segment_inv_pre_compress = async (o: {
126+ array: Int16Array;
127+ quant_scale_factor: number;
128+ segment_length: number;
129+ dtype: "int16" | "float32";
130+ num_samples: number;
131+ num_channels: number;
132+}): Promise<ArrayBuffer> => {
133+ const {
134+ array,
135+ quant_scale_factor,
136+ segment_length,
137+ dtype,
138+ num_samples,
139+ num_channels,
140+ } = o;
141+ if (segment_length > 0 && segment_length < num_samples) {
142+ const segment_ranges = _get_segment_ranges(num_samples, segment_length);
143+ const prepared_segments = await Promise.all(
144+ segment_ranges.map(
145+ async (segment_range) =>
146+ await qfc_inv_pre_compress({
147+ array: array.slice(
148+ segment_range[0] * num_channels,
149+ segment_range[1] * num_channels,
150+ ),
151+ quant_scale_factor,
152+ dtype,
153+ num_samples: segment_range[1] - segment_range[0],
154+ num_channels,
155+ }),
156+ ),
157+ );
158+ if (dtype === "int16") {
159+ return concatenateInt16Arrays(prepared_segments as Int16Array[]);
160+ } else if (dtype === "float32") {
161+ return concatenateFloat32Arrays(prepared_segments as Float32Array[]);
162+ } else {
163+ throw Error(`Unexpected dtype: ${dtype}`);
164+ }
165+ } else {
166+ return await qfc_inv_pre_compress({
167+ array,
168+ quant_scale_factor,
169+ dtype,
170+ num_samples,
171+ num_channels,
172+ });
173+ }
174+};
175+
176+const _get_segment_ranges = (
177+ total_length: number,
178+ segment_length: number,
179+): [number, number][] => {
180+ const segment_ranges: [number, number][] = [];
181+ for (
182+ let start_index = 0;
183+ start_index < total_length;
184+ start_index += segment_length
185+ ) {
186+ segment_ranges.push([
187+ start_index,
188+ Math.min(start_index + segment_length, total_length),
189+ ]);
190+ }
191+ const size_of_final_segment =
192+ segment_ranges[segment_ranges.length - 1][1] -
193+ segment_ranges[segment_ranges.length - 1][0];
194+ const half_segment_length = Math.floor(segment_length / 2);
195+ if (
196+ size_of_final_segment < half_segment_length &&
197+ segment_ranges.length > 1
198+ ) {
199+ const adjustment = half_segment_length - size_of_final_segment;
200+ segment_ranges[segment_ranges.length - 2] = [
201+ segment_ranges[segment_ranges.length - 2][0],
202+ segment_ranges[segment_ranges.length - 2][1] - adjustment,
203+ ];
204+ segment_ranges[segment_ranges.length - 1] = [
205+ segment_ranges[segment_ranges.length - 1][0] - adjustment,
206+ segment_ranges[segment_ranges.length - 1][1],
207+ ];
208+ }
209+ return segment_ranges;
210+};
211+
212+const qfc_inv_pre_compress = async (o: {
213+ array: Int16Array;
214+ quant_scale_factor: number;
215+ dtype: "int16" | "float32";
216+ num_samples: number;
217+ num_channels: number;
218+}): Promise<Int16Array | Float32Array> => {
219+ const { array, quant_scale_factor, dtype, num_samples, num_channels } = o;
220+
221+ const m = Math.floor(num_samples / 2);
222+ const isEvenNumberOfSamples = num_samples % 2 === 0;
223+ const qs = quant_scale_factor;
224+ const x_re = new Float32Array((m + 1) * num_channels);
225+ for (let i = 0; i < m + 1; i++) {
226+ for (let j = 0; j < num_channels; j++) {
227+ x_re[i * num_channels + j] = array[i * num_channels + j] / qs;
228+ }
229+ }
230+ // ns - (ns // 2 + 1) + 2 = ns - ns // 2 - 1 + 2 = ns - ns // 2 + 1 = ns // 2 + 1
231+ const x_im = new Float32Array((m + 1) * num_channels);
232+ x_im.fill(0); // probably not necessary
233+ const mm = isEvenNumberOfSamples ? m : m + 1;
234+ for (let i = 1; i < mm; i++) {
235+ for (let j = 0; j < num_channels; j++) {
236+ x_im[i * num_channels + j] =
237+ array[(m + 1 + (i - 1)) * num_channels + j] / qs;
238+ }
239+ }
240+
241+ const x_fft = await irfftMultiChannel(x_re, x_im, num_samples, num_channels);
242+ for (let i = 0; i < x_fft.length; i++) {
243+ x_fft[i] = x_fft[i] * Math.sqrt(num_samples);
244+ }
245+ if (dtype === "int16") {
246+ const ret = new Int16Array(x_fft.byteLength);
247+ for (let i = 0; i < x_fft.length; i++) {
248+ ret[i] = Math.round(x_fft[i]);
249+ }
250+ return ret;
251+ } else if (dtype === "float32") {
252+ return x_fft;
253+ } else {
254+ throw Error(`Unexpected dtype: ${dtype}`);
255+ }
256+};
257+
258+const irfftMultiChannel = async (
259+ x_re: Float32Array,
260+ x_im: Float32Array,
261+ num_samples: number,
262+ num_channels: number,
263+): Promise<Float32Array> => {
264+ const ns = num_samples;
265+ const nc = num_channels;
266+ const ret = new Float32Array(ns * nc);
267+ for (let j = 0; j < nc; j++) {
268+ const a_re = new Float32Array(x_re.length / nc);
269+ const a_im = new Float32Array(x_im.length / nc);
270+ for (let i = 0; i < x_re.length / num_channels; i++) {
271+ a_re[i] = x_re[i * nc + j];
272+ a_im[i] = x_im[i * nc + j];
273+ }
274+ const b = await irfft(a_re, a_im, num_samples);
275+ for (let i = 0; i < ns; i++) {
276+ ret[i * nc + j] = b[i];
277+ }
278+ }
279+ return ret;
280+};
281+
282+const irfft = async (
283+ x_re: Float32Array,
284+ x_im: Float32Array,
285+ num_samples: number,
286+): Promise<Float32Array> => {
287+ if (x_re.length != Math.floor(num_samples / 2) + 1) {
288+ throw Error(
289+ `Unexpected x_re length. Expected ${Math.floor(num_samples / 2) + 1}, got ${x_re.length}`,
290+ );
291+ }
292+ if (x_im.length != Math.floor(num_samples / 2) + 1) {
293+ throw Error(
294+ `Unexpected x_im length. Expected ${Math.floor(num_samples / 2) + 1}, got ${x_im.length}`,
295+ );
296+ }
297+ const x_re_copy = new Float32Array(num_samples);
298+ const x_im_copy = new Float32Array(num_samples);
299+ for (let i = 0; i < x_re.length; i++) {
300+ x_re_copy[i] = x_re[i];
301+ x_im_copy[i] = x_im[i];
302+ if (i > 0) {
303+ // the last case is i = x_re.length - 1
304+ // in this case we are filling in (num_samples - x_re.length + 1)
305+ // x_re.length = num_samples // 2 + 1
306+ // so i = num_samples // 2
307+ // and we're filling in (num_samples - num_samples // 2)
308+ // in the case where num_samples is even, this is num_samples // 2, and imag part is zero there, so it's correct
309+ // in the case where num_samples is odd, this is num_samples // 2 + 1, which is correct
310+ x_re_copy[num_samples - i] = x_re[i];
311+ x_im_copy[num_samples - i] = -x_im[i];
312+ }
313+ }
314+ inverseTransform(x_re_copy, x_im_copy); // in place
315+ // let's verify that imaginary part is close to zero
316+ for (let i = 0; i < num_samples; i++) {
317+ if (Math.abs(x_im_copy[i]) > 1e-5) {
318+ throw Error("Unexpected non-zero imaginary part after inverse transform");
319+ }
320+ }
321+ return x_re_copy;
322+};
323+
324+const concatenateInt16Arrays = (arrays: Int16Array[]): ArrayBuffer => {
325+ const total_length = arrays.reduce((acc, x) => acc + x.length, 0);
326+ const concatenated = new Int16Array(total_length);
327+ let offset = 0;
328+ for (const a of arrays) {
329+ concatenated.set(a, offset);
330+ offset += a.length;
331+ }
332+ return concatenated.buffer;
333+};
334+
335+const concatenateFloat32Arrays = (arrays: Float32Array[]): ArrayBuffer => {
336+ const total_length = arrays.reduce((acc, x) => acc + x.length, 0);
337+ const concatenated = new Float32Array(total_length);
338+ let offset = 0;
339+ for (const a of arrays) {
340+ concatenated.set(a, offset);
341+ offset += a.length;
342+ }
343+ return concatenated.buffer;
344+};
src/remote-h5-file/lib/lindi/zarrDecodeChunkArray.tsadded+240−0View file
@@ -0,0 +1,240 @@
1+import { Blosc } from "numcodecs";
2+import pako from "pako";
3+import { qfcDecompress } from "./qfc";
4+
5+/* eslint-disable @typescript-eslint/no-explicit-any */
6+const zarrDecodeChunkArray = async (
7+ chunk: ArrayBuffer,
8+ dtype?: string,
9+ compressor?: any,
10+ filters?: any[],
11+ shape?: number[],
12+): Promise<any> => {
13+ let ret: any = chunk;
14+ if (compressor) {
15+ if (compressor.id === "blosc") {
16+ ret = await new Blosc().decode(chunk);
17+ } else if (compressor.id === "gzip") {
18+ ret = pako.inflate(chunk);
19+ } else if (compressor.id === "neurosift.mp4") {
20+ // ret = await decodeMp4(chunk, shape![0], shape![1], shape![2]);
21+ throw Error("neurosift.mp4 decoder not yet implemented");
22+ } else if (compressor.id === "qfc") {
23+ if (!shape) {
24+ throw Error("No shape for qfc");
25+ }
26+ ret = await qfcDecompress(chunk, shape, compressor);
27+ } else {
28+ throw Error("Unhandled compressor " + compressor.id);
29+ }
30+ }
31+ // check if Uint8Array
32+ if (ret instanceof Uint8Array) {
33+ ret = ret.buffer;
34+ }
35+ if (dtype === "|O") {
36+ if (!shape) throw Error("No shape for |O");
37+ if (!filters) {
38+ throw Error("No filters for |O");
39+ }
40+ if (filters.length === 0) {
41+ throw Error("No filters for |O");
42+ }
43+ // work backwards through the filters, besides the first one which should be json2
44+ for (let i = filters.length - 1; i > 0; i--) {
45+ ret = await applyFilter(ret, filters[i]);
46+ }
47+ const filter0 = filters[0];
48+ if (filter0.id !== "json2") {
49+ throw Error("First filter for |O should be json2");
50+ }
51+ ret = await applyFilterToOType(ret, filter0, shape);
52+ } else {
53+ // work our way backward through the filters
54+ if (filters) {
55+ for (let i = filters.length - 1; i >= 0; i--) {
56+ ret = await applyFilter(ret, filters[i]);
57+ }
58+ }
59+ if (!dtype) {
60+ // pass
61+ } else if (dtype === "<f4") {
62+ ret = new Float32Array(ret);
63+ } else if (dtype === "<f8") {
64+ ret = new Float64Array(ret);
65+ } else if (dtype === "<i1" || dtype === "|i1") {
66+ ret = new Int8Array(ret);
67+ } else if (dtype === "<i2") {
68+ ret = new Int16Array(ret);
69+ } else if (dtype === "<i4") {
70+ ret = new Int32Array(ret);
71+ } else if (dtype === "<i8") {
72+ const ret0 = new BigInt64Array(ret);
73+ // convert to Int32Array because javascript has trouble mixing BigInt64Array with other types
74+ ret = new Int32Array(ret0.length);
75+ for (let i = 0; i < ret0.length; i++) {
76+ ret[i] = Number(ret0[i]);
77+ }
78+ } else if (dtype === "<u1" || dtype === "|u1") {
79+ ret = new Uint8Array(ret);
80+ } else if (dtype === "<u2") {
81+ ret = new Uint16Array(ret);
82+ } else if (dtype === "<u4") {
83+ ret = new Uint32Array(ret);
84+ } else if (dtype === "<u8") {
85+ const ret0 = new BigUint64Array(ret);
86+ // convert to Uint32Array because javascript has trouble mixing BigUint64Array with other types
87+ ret = new Uint32Array(ret0.length);
88+ for (let i = 0; i < ret0.length; i++) {
89+ ret[i] = Number(ret0[i]);
90+ }
91+ } else if (dtype === "|b1") {
92+ ret = new Uint8Array(ret);
93+ } else if (dtype.startsWith("<U")) {
94+ const fixedLength = parseInt(dtype.slice(2));
95+ const nn = ret.byteLength / (fixedLength * 4);
96+ const ret2 = [];
97+ for (let i = 0; i < nn; i++) {
98+ const ret1 = new Uint32Array(ret, i * fixedLength * 4, fixedLength);
99+ const ret3 = [];
100+ for (let j = 0; j < fixedLength; j++) {
101+ if (ret1[j] === 0) break; // null terminates the string, matching NumPy behavior
102+ ret3.push(String.fromCodePoint(ret1[j]));
103+ }
104+ ret2.push(ret3.join(""));
105+ }
106+ ret = ret2;
107+ } else if (dtype.startsWith("|S")) {
108+ const fixedLength = parseInt(dtype.slice(2));
109+ const nn = ret.byteLength / fixedLength;
110+ const ret2 = [];
111+ for (let i = 0; i < nn; i++) {
112+ const ret1 = new Uint8Array(ret, i * fixedLength, fixedLength);
113+ // Find first null byte to trim, matching NumPy behavior
114+ let len = fixedLength;
115+ for (let j = 0; j < fixedLength; j++) {
116+ if (ret1[j] === 0) {
117+ len = j;
118+ break;
119+ }
120+ }
121+ ret2.push(new TextDecoder().decode(ret1.subarray(0, len)));
122+ }
123+ ret = ret2;
124+ } else {
125+ throw Error("Unhandled dtype " + dtype);
126+ }
127+ }
128+ return ret;
129+};
130+
131+const applyFilterToOType = async (
132+ chunk: ArrayBuffer,
133+ filter: any,
134+ shape: number[],
135+) => {
136+ if (filter.id === "vlen-utf8") {
137+ const view = new DataView(chunk);
138+ const ret = [];
139+ let i = 4;
140+ while (i < chunk.byteLength) {
141+ const byte1 = view.getUint32(i, true);
142+ const byte2 = view.getUint32(i + 1, true);
143+ const byte3 = view.getUint32(i + 2, true);
144+ const byte4 = view.getUint32(i + 3, true);
145+ const len = byte1 + (byte2 << 8) + (byte3 << 16) + (byte4 << 24);
146+ i += 4;
147+ ret.push(new TextDecoder().decode(chunk.slice(i, i + len)));
148+ i += len;
149+ }
150+ return ret;
151+ } else if (filter.id === "vlen-bytes") {
152+ const view = new DataView(chunk);
153+ const ret = [];
154+ let i = 4;
155+ while (i < chunk.byteLength) {
156+ const byte1 = view.getUint32(i, true);
157+ const byte2 = view.getUint32(i + 1, true);
158+ const byte3 = view.getUint32(i + 2, true);
159+ const byte4 = view.getUint32(i + 3, true);
160+ const len = byte1 + (byte2 << 8) + (byte3 << 16) + (byte4 << 24);
161+ i += 4;
162+ ret.push(chunk.slice(i, i + len));
163+ i += len;
164+ }
165+ return ret;
166+ } else if (filter.id === "json2") {
167+ const aa = JSON.parse(new TextDecoder().decode(chunk));
168+ // aa has the form [item1, item2, ..., itemN, '|O', shape]
169+ if (aa.length <= 2) {
170+ console.warn("Unexpected json2", aa);
171+ return new TextDecoder().decode(chunk);
172+ }
173+ if (!sameShape(aa[aa.length - 1], shape)) {
174+ throw Error(
175+ `Unexpected shape for json2 filter: ${aa[aa.length - 1]} !== ${shape}`,
176+ );
177+ }
178+ if (!(aa[aa.length - 2] === "|O")) {
179+ throw Error(
180+ `Unexpected dtype for json2 filter: ${aa[aa.length - 2]} !== |O`,
181+ );
182+ }
183+ if (!(aa.length - 2 === shape[0])) {
184+ throw Error(
185+ `Unexpected length for json2 filter: ${aa.length - 2} !== ${shape[0]}`,
186+ );
187+ }
188+ if (shape.length > 1) {
189+ return flattenArray(aa.slice(0, aa.length - 2), shape);
190+ }
191+ return aa.slice(0, aa.length - 2);
192+ } else {
193+ throw Error("Unhandled filter for |O " + filter.id);
194+ }
195+};
196+
197+const applyFilter = async (chunk: ArrayBuffer, filter: any) => {
198+ if (filter.id === "zlib") {
199+ const a = pako.inflate(chunk);
200+ return a.buffer;
201+ } else if (filter.id === "blosc") {
202+ return new Blosc().decode(chunk);
203+ } else if (filter.id === "shuffle") {
204+ const elementSize = filter.elementsize;
205+ const view = new DataView(chunk);
206+ const ret = new Uint8Array(chunk.byteLength);
207+ const a = chunk.byteLength / elementSize;
208+ for (let i = 0; i < chunk.byteLength; i++) {
209+ const b = i % elementSize;
210+ const c = Math.floor(i / elementSize) * elementSize;
211+ const j = b * a + c;
212+ ret[j] = view.getUint8(i);
213+ }
214+ return ret.buffer;
215+ }
216+ console.warn("Filter not yet implemented", filter);
217+ throw Error("Filter not yet implemented");
218+};
219+
220+const sameShape = (a: number[], b: number[]): boolean => {
221+ if (a.length !== b.length) return false;
222+ for (let i = 0; i < a.length; i++) {
223+ if (a[i] !== b[i]) return false;
224+ }
225+ return true;
226+};
227+
228+const flattenArray = (aa: any[], shape: number[]): any[] => {
229+ if (shape.length === 1) return aa;
230+ const ret = [];
231+ for (let i = 0; i < shape[0]; i++) {
232+ const x = flattenArray(aa[i], shape.slice(1));
233+ for (const xx of x) {
234+ ret.push(xx);
235+ }
236+ }
237+ return ret;
238+};
239+
240+export default zarrDecodeChunkArray;
src/remote-h5-file/lib/numcodecs.d.tsadded+5−0View file
@@ -0,0 +1,5 @@
1+declare module "numcodecs" {
2+ export class Blosc {
3+ decode: (data: ArrayBuffer) => Promise<ArrayBuffer>;
4+ }
5+}
tsconfig.app.jsonadded+25−0View file
@@ -0,0 +1,25 @@
1+{
2+ "compilerOptions": {
3+ "tsBuildInfoFile": "./node_modules/.tmp/tsconfig.app.tsbuildinfo",
4+ "target": "ES2022",
5+ "useDefineForClassFields": true,
6+ "lib": ["ES2022", "DOM", "DOM.Iterable"],
7+ "module": "ESNext",
8+ "types": ["vite/client"],
9+ "skipLibCheck": true,
10+
11+ "moduleResolution": "bundler",
12+ "allowImportingTsExtensions": true,
13+ "moduleDetection": "force",
14+ "noEmit": true,
15+ "jsx": "react-jsx",
16+ "resolveJsonModule": true,
17+
18+ "strict": true,
19+ "noUnusedLocals": false,
20+ "noUnusedParameters": false,
21+ "noFallthroughCasesInSwitch": true,
22+ "noUncheckedSideEffectImports": true
23+ },
24+ "include": ["src"]
25+}
tsconfig.jsonadded+7−0View file
@@ -0,0 +1,7 @@
1+{
2+ "files": [],
3+ "references": [
4+ { "path": "./tsconfig.app.json" },
5+ { "path": "./tsconfig.node.json" }
6+ ]
7+}
tsconfig.node.jsonadded+18−0View file
@@ -0,0 +1,18 @@
1+{
2+ "compilerOptions": {
3+ "tsBuildInfoFile": "./node_modules/.tmp/tsconfig.node.tsbuildinfo",
4+ "target": "ES2023",
5+ "lib": ["ES2023"],
6+ "module": "ESNext",
7+ "types": [],
8+ "skipLibCheck": true,
9+
10+ "moduleResolution": "bundler",
11+ "allowImportingTsExtensions": true,
12+ "moduleDetection": "force",
13+ "noEmit": true,
14+
15+ "strict": true
16+ },
17+ "include": ["vite.config.ts"]
18+}
vite.config.tsadded+8−0View file
@@ -0,0 +1,8 @@
1+import { defineConfig } from "vite";
2+import react from "@vitejs/plugin-react";
3+
4+// https://vite.dev/config/
5+export default defineConfig({
6+ plugins: [react()],
7+ base: "/remote-hdf5-lazy-read/",
8+});