1# Lazy reading of remote HDF5 / NWB files
3This is a small demonstration of how [neurosift](https://neurosift.app/) browses
4large [NWB](https://www.nwb.org/) files directly in the browser, pulling data out
5of HDF5 files that live on remote storage without downloading them in full and
6without any backend server.
8The [live demo](?#demo) opens one NWB file from
9[DANDI dandiset 000986](https://dandiarchive.org/dandiset/000986) (recordings from
10mouse auditory cortex) straight from DANDI's S3 bucket. It reads the session
11metadata, walks the top of the file's group structure, and pulls a short window
12out of a multi-million-sample timeseries, all on demand.
14## Why this is possible
16An HDF5 file is not a blob you have to read start to finish. It is a small amount
17of structural metadata (a superblock, some B-trees, object headers) together with
18the array data laid out in independently addressable chunks. If you know which
19byte ranges hold the thing you want, you can read just those bytes and ignore the
20rest.
22That maps neatly onto an HTTP feature that has been around forever: the `Range`
23request. DANDI's S3 objects honor range requests, so a browser can treat a remote
24multi-gigabyte NWB file as if it were local, fetching a few kilobytes here and
25there as it goes. Opening the file, expanding a group, or slicing a dataset each
26turn into a handful of small requests rather than a download.
28There are two ways neurosift does the reading, and the demo runs both next to each
29other so you can compare them.
31## Reading the HDF5 directly
33The first approach reads the actual HDF5 file. The HDF5 C library is compiled to
34WebAssembly ([h5wasm](https://github.com/usnistgov/h5wasm)) and run inside a web
35worker. The worker hands h5wasm a file that is backed by the network rather than
36by disk, using emscripten's lazy filesystem:
38```js
39// remote-h5-worker
40FS.createLazyFile('/', fname, url, true, false, headers, chunkSize);
41const file = new h5wasm.File(fname);
42```
44Whenever h5wasm tries to read some offset in that file, the lazy filesystem
45fetches the chunk that contains it over HTTP and caches it. So the parsing is done
46by the real HDF5 library (the same code you would run locally), but the bytes
47trickle in from S3 as the library walks the structure. The worker exposes three
48calls, `getGroup`, `getDataset`, and `getDatasetData(path, { slice })`, and the
49main thread talks to it through a thin wrapper (`RemoteH5File`) that caches
50results.
52The catch is latency. Walking HDF5's B-trees can take a lot of small round trips
53before you have the metadata you need, and over a network that adds up.
55## Reading through a LINDI index
57The second approach side-steps that latency.
58[LINDI](https://github.com/neurodatawithoutborders/lindi) precomputes the answer
59to "where is everything?" once, on the server, and stores it as a single JSON
60file. The format follows the [kerchunk](https://github.com/fsspec/kerchunk)
61convention and is, in fact, a valid Zarr store. It is a dictionary of `refs` whose
62keys are Zarr paths and whose values are one of:
64```jsonc
65{
66 "refs": {
67 "units/.zgroup": "{\"zarr_format\":2}", // small data stored inline
68 "units/spike_times/.zarray": { /* shape, dtype, compressor, ... */ },
69 "units/spike_times/0": ["<original-hdf5-url>", 12345, 678] // [url, offset, length]
70 }
71}
72```
74Group structure, attributes, and small datasets are inlined, while large array
75chunks are left as `[url, offset, length]` references that point straight back
76into the original HDF5 file on S3. LINDI also defines a few extra Zarr annotations
77(`_SCALAR`, `_REFERENCE`, `_COMPOUND_DTYPE`, `_EXTERNAL_ARRAY_LINK`) so it can
78faithfully represent HDF5 features that plain Zarr has no notion of, such as scalar
79datasets, object references, and compound types.
81The payoff is that the entire structure of the file arrives in one request. After
82that, reading actual data still uses range requests against the original HDF5,
83with the chunks decoded client-side (blosc, zlib, and friends). neurosift keeps
84pre-generated indexes for published dandisets at `lindi.neurosift.org` and uses
85one when it is available, falling back to direct h5wasm reading when it is not.
87## Putting it together
89```
90NwbPage ── hdf5Interface ──► RemoteH5File ─RPC→ worker ─► h5wasm ─Range→ S3
91 └► RemoteH5FileLindi ──► JSON index, then Range→ S3
92```
94A DANDI asset URL is first followed to its underlying S3 object. neurosift then
95prefers the LINDI index if one exists and otherwise reads the raw HDF5 through the
96h5wasm worker. Either way, the viewers downstream only ask for the slices they
97actually draw, which is what keeps even very large files responsive.
99The reading library under `src/remote-h5-file` is taken unchanged from neurosift,
100and the worker is loaded from `tempory.net/js/RemoteH5Worker.js`, built from
101[magland/remote-h5-worker](https://github.com/magland/remote-h5-worker).
103## About the demo
105The two panels read the same file using the two strategies above. Each one reports
106how long the open took, lists the root group, shows a handful of session and
107subject fields, and then loads the first 30,000 of roughly 7.5 million samples of
108the pupil-diameter and running-speed traces (well under one percent of the data)
109before plotting them. It finishes by reading the trials table and plotting stimulus
110frequency across the session. If you open the network tab while it runs, you can
111watch the partial range requests come back.
113## Running locally
115```bash
116npm install
117npm run dev
118```
120## Credits
122Built on [neurosift](https://github.com/flatironinstitute/neurosift),
123[h5wasm](https://github.com/usnistgov/h5wasm), and
124[LINDI](https://github.com/neurodatawithoutborders/lindi). Example data is
125[DANDI:000986](https://dandiarchive.org/dandiset/000986).