Lazy reading of remote HDF5 / NWB files#

This is a small demonstration of how neurosift browses large NWB files directly in the browser, pulling data out of HDF5 files that live on remote storage without downloading them in full and without any backend server.

The source code for this demo is at concept-collection/remote-hdf5-lazy-read.

The live demo opens one NWB file from DANDI dandiset 000986 (recordings from mouse auditory cortex) straight from DANDI's S3 bucket. It reads the session metadata, walks the top of the file's group structure, and pulls a short window out of a multi-million-sample timeseries, all on demand.

Why this is possible#

An HDF5 file is not a blob you have to read start to finish. It is a small amount of structural metadata (a superblock, some B-trees, object headers) together with the array data laid out in independently addressable chunks. If you know which byte ranges hold the thing you want, you can read just those bytes and ignore the rest.

That maps neatly onto an HTTP feature that has been around forever: the Range request. DANDI's S3 objects honor range requests, so a browser can treat a remote multi-gigabyte NWB file as if it were local, fetching a few kilobytes here and there as it goes. Opening the file, expanding a group, or slicing a dataset each turn into a handful of small requests rather than a download.

There are two ways neurosift does the reading, and the demo runs both next to each other so you can compare them.

Reading the HDF5 directly#

The first approach reads the actual HDF5 file. The HDF5 C library is compiled to WebAssembly (h5wasm) and run inside a web worker. The worker hands h5wasm a file that is backed by the network rather than by disk, using emscripten's lazy filesystem:

// remote-h5-worker
FS.createLazyFile('/', fname, url, true, false, headers, chunkSize);
const file = new h5wasm.File(fname);

Whenever h5wasm tries to read some offset in that file, the lazy filesystem fetches the chunk that contains it over HTTP and caches it. So the parsing is done by the real HDF5 library (the same code you would run locally), but the bytes trickle in from S3 as the library walks the structure. The worker exposes three calls, getGroup, getDataset, and getDatasetData(path, { slice }), and the main thread talks to it through a thin wrapper (RemoteH5File) that caches results.

The catch is latency. Walking HDF5's B-trees can take a lot of small round trips before you have the metadata you need, and over a network that adds up.

Reading through a LINDI index#

The second approach side-steps that latency. LINDI precomputes the answer to "where is everything?" once, on the server, and stores it as a single JSON file. The format follows the kerchunk convention and is, in fact, a valid Zarr store. It is a dictionary of refs whose keys are Zarr paths and whose values are one of:

{
  "refs": {
    "units/.zgroup": "{\"zarr_format\":2}",      // small data stored inline
    "units/spike_times/.zarray": { /* shape, dtype, compressor, ... */ },
    "units/spike_times/0": ["<original-hdf5-url>", 12345, 678]  // [url, offset, length]
  }
}

Group structure, attributes, and small datasets are inlined, while large array chunks are left as [url, offset, length] references that point straight back into the original HDF5 file on S3. LINDI also defines a few extra Zarr annotations (_SCALAR, _REFERENCE, _COMPOUND_DTYPE, _EXTERNAL_ARRAY_LINK) so it can faithfully represent HDF5 features that plain Zarr has no notion of, such as scalar datasets, object references, and compound types.

The payoff is that the entire structure of the file arrives in one request. After that, reading actual data still uses range requests against the original HDF5, with the chunks decoded client-side (blosc, zlib, and friends). neurosift keeps pre-generated indexes for published dandisets at lindi.neurosift.org and uses one when it is available, falling back to direct h5wasm reading when it is not.

Putting it together#

NwbPage ── hdf5Interface ──► RemoteH5File       ─RPC→ worker ─► h5wasm ─Range→ S3
                          └► RemoteH5FileLindi ──► JSON index, then Range→ S3

A DANDI asset URL is first followed to its underlying S3 object. neurosift then prefers the LINDI index if one exists and otherwise reads the raw HDF5 through the h5wasm worker. Either way, the viewers downstream only ask for the slices they actually draw, which is what keeps even very large files responsive.

The reading library under src/remote-h5-file is taken unchanged from neurosift, and the worker is loaded from tempory.net/js/RemoteH5Worker.js, built from magland/remote-h5-worker.

About the demo#

The two panels read the same file using the two strategies above. Each one reports how long the open took, lists the root group, shows a handful of session and subject fields, and then loads the first 30,000 of roughly 7.5 million samples of the pupil-diameter and running-speed traces (well under one percent of the data) before plotting them. It finishes by reading the trials table and plotting stimulus frequency across the session. If you open the network tab while it runs, you can watch the partial range requests come back.

Running locally#

npm install
npm run dev

Credits#

Built on neurosift, h5wasm, and LINDI. Example data is DANDI:000986.