d9fc1b2Demonstrate lazy reading of remote HDF5/NWB files (direct h5wasm + LINDI)magland 1# Lazy reading of remote HDF5 / NWB files
3This is a small demonstration of how [neurosift](https://neurosift.app/) browses
4large [NWB](https://www.nwb.org/) files directly in the browser, pulling data out
5of HDF5 files that live on remote storage without downloading them in full and
6without any backend server.
9[concept-collection/remote-hdf5-lazy-read](https://github.com/concept-collection/remote-hdf5-lazy-read).
d9fc1b2Demonstrate lazy reading of remote HDF5/NWB files (direct h5wasm + LINDI)magland 11The [live demo](?#demo) opens one NWB file from
12[DANDI dandiset 000986](https://dandiarchive.org/dandiset/000986) (recordings from
13mouse auditory cortex) straight from DANDI's S3 bucket. It reads the session
14metadata, walks the top of the file's group structure, and pulls a short window
15out of a multi-million-sample timeseries, all on demand.
17## Why this is possible
19An HDF5 file is not a blob you have to read start to finish. It is a small amount
20of structural metadata (a superblock, some B-trees, object headers) together with
21the array data laid out in independently addressable chunks. If you know which
22byte ranges hold the thing you want, you can read just those bytes and ignore the
23rest.
25That maps neatly onto an HTTP feature that has been around forever: the `Range`
26request. DANDI's S3 objects honor range requests, so a browser can treat a remote
27multi-gigabyte NWB file as if it were local, fetching a few kilobytes here and
28there as it goes. Opening the file, expanding a group, or slicing a dataset each
29turn into a handful of small requests rather than a download.
31There are two ways neurosift does the reading, and the demo runs both next to each
32other so you can compare them.
34## Reading the HDF5 directly
36The first approach reads the actual HDF5 file. The HDF5 C library is compiled to
37WebAssembly ([h5wasm](https://github.com/usnistgov/h5wasm)) and run inside a web
38worker. The worker hands h5wasm a file that is backed by the network rather than
39by disk, using emscripten's lazy filesystem:
41```js
42// remote-h5-worker
43FS.createLazyFile('/', fname, url, true, false, headers, chunkSize);
44const file = new h5wasm.File(fname);
45```
47Whenever h5wasm tries to read some offset in that file, the lazy filesystem
48fetches the chunk that contains it over HTTP and caches it. So the parsing is done
49by the real HDF5 library (the same code you would run locally), but the bytes
50trickle in from S3 as the library walks the structure. The worker exposes three
51calls, `getGroup`, `getDataset`, and `getDatasetData(path, { slice })`, and the
52main thread talks to it through a thin wrapper (`RemoteH5File`) that caches
53results.
55The catch is latency. Walking HDF5's B-trees can take a lot of small round trips
56before you have the metadata you need, and over a network that adds up.
58## Reading through a LINDI index
60The second approach side-steps that latency.
61[LINDI](https://github.com/neurodatawithoutborders/lindi) precomputes the answer
62to "where is everything?" once, on the server, and stores it as a single JSON
63file. The format follows the [kerchunk](https://github.com/fsspec/kerchunk)
64convention and is, in fact, a valid Zarr store. It is a dictionary of `refs` whose
65keys are Zarr paths and whose values are one of:
67```jsonc
68{
69 "refs": {
70 "units/.zgroup": "{\"zarr_format\":2}", // small data stored inline
71 "units/spike_times/.zarray": { /* shape, dtype, compressor, ... */ },
72 "units/spike_times/0": ["<original-hdf5-url>", 12345, 678] // [url, offset, length]
73 }
74}
75```
77Group structure, attributes, and small datasets are inlined, while large array
78chunks are left as `[url, offset, length]` references that point straight back
79into the original HDF5 file on S3. LINDI also defines a few extra Zarr annotations
80(`_SCALAR`, `_REFERENCE`, `_COMPOUND_DTYPE`, `_EXTERNAL_ARRAY_LINK`) so it can
81faithfully represent HDF5 features that plain Zarr has no notion of, such as scalar
82datasets, object references, and compound types.
84The payoff is that the entire structure of the file arrives in one request. After
85that, reading actual data still uses range requests against the original HDF5,
86with the chunks decoded client-side (blosc, zlib, and friends). neurosift keeps
87pre-generated indexes for published dandisets at `lindi.neurosift.org` and uses
88one when it is available, falling back to direct h5wasm reading when it is not.
90## Putting it together
92```
93NwbPage ── hdf5Interface ──► RemoteH5File ─RPC→ worker ─► h5wasm ─Range→ S3
94 └► RemoteH5FileLindi ──► JSON index, then Range→ S3
95```
97A DANDI asset URL is first followed to its underlying S3 object. neurosift then
98prefers the LINDI index if one exists and otherwise reads the raw HDF5 through the
99h5wasm worker. Either way, the viewers downstream only ask for the slices they
100actually draw, which is what keeps even very large files responsive.
102The reading library under `src/remote-h5-file` is taken unchanged from neurosift,
103and the worker is loaded from `tempory.net/js/RemoteH5Worker.js`, built from
104[magland/remote-h5-worker](https://github.com/magland/remote-h5-worker).
106## About the demo
108The two panels read the same file using the two strategies above. Each one reports
109how long the open took, lists the root group, shows a handful of session and
110subject fields, and then loads the first 30,000 of roughly 7.5 million samples of
111the pupil-diameter and running-speed traces (well under one percent of the data)
112before plotting them. It finishes by reading the trials table and plotting stimulus
113frequency across the session. If you open the network tab while it runs, you can
114watch the partial range requests come back.
116## Running locally
118```bash
119npm install
120npm run dev
121```
123## Credits
125Built on [neurosift](https://github.com/flatironinstitute/neurosift),
126[h5wasm](https://github.com/usnistgov/h5wasm), and
127[LINDI](https://github.com/neurodatawithoutborders/lindi). Example data is
128[DANDI:000986](https://dandiarchive.org/dandiset/000986).