challenge-tools#

Shared tooling for the instrument simulation challenges: the submission schema, validation of submissions, the error measures, the reference-selection rule, and the generator of the static comparison site. Every challenge repository (for example challenges/string and challenges/plate) installs this package in its workflow and runs it against the solution repositories listed in its registry. Solution repositories run only the validator.

Install#

pip install git+https://vault1.magland.org/challenges/challenge-tools

Commands#

challenge-tools fetch    --registry registry/solution-repos.json --clones clones
challenge-tools validate --challenges challenges --registry registry/solution-repos.json --clones clones
challenge-tools build    --challenges challenges --registry registry/solution-repos.json --clones clones --out site
challenge-tools validate --challenges <spec-repo>/challenges --solutions solutions      # a solution repository checking itself

fetch clones or updates every registered repository (shallow, one branch, anonymously; git lfs pull is run when git-lfs is available). validate checks each submission JSON against schema.json, the wav files against the challenge's output specification, and any field exports against the challenge's recommended grid. build validates, computes every pairwise error, chooses the reference, and writes the site.

What a challenge must provide#

A directory challenges/<challenge-id>/ with a README.md (the specification) and a challenge.json with at least:

{
  "challenge_id": "...",
  "output": {"sample_rate_hz": 48000, "num_samples": 96000, "channels": 1, "sample_format": "float32", "signal_scale": 1.0},
  "metrics": {"primary": "waveform_l2", "waveform_l2": {}, "spectrogram_l2": {"n_fft": 2048, "hop": 512}},
  "fields": {"format": "hdf5", "axes": ["t", "x"], "shape": [4001, 201]},
  "reference_solution": null
}

fields is optional and describes the grid of the optional field export: datasets named after axes plus a dataset u of the given shape. reference_solution overrides the automatic reference with a repo/submission/solution key.

Scoring and reference selection#

The error measures live in tools/challenge_tools/metrics.py and the reference rule in tools/challenge_tools/reference.py. The challenge READMEs are the specification; this code is intended to follow them exactly. The rule: within a submission, more_accurate_than asserts a partial order; candidates for reference are the maximal elements of each submission's order; a candidate is scored by the fraction of asserted pairs from other submissions whose order it reproduces under the primary error; ties are broken by the candidate's convergence gap (error to its nearest asserted-worse sibling) and then by mean error to the other candidates.

Playback#

The wav files are exact signals with a loud strike transient and a much quieter ring, and they are at physical scale, so playing them raw is either faint or, with a gain, a hearing hazard on headphones. The sites use tools/challenge_tools/player.js, a Web Audio player built for safety first: each file is loudness-normalized at load time so that its loudest 400 ms window sits at -16 dBFS rms (about the loudness of a streamed music track), a fast limiter holds the strike transient at -10 dBFS, and a hard clipper at -6 dBFS is the last stage of the graph and cannot be bypassed, so nothing louder than a quiet music peak reaches the output whatever the settings. The first playback on a page asks for confirmation. The listener adjusts a bounded level offset (-20 to +6 dB, starting at -12 dB in a new browser and remembered there); raising it above the normalized level, or switching the limiter off, asks for confirmation. The player also draws a spectrogram of each file on request (Hann 2048, hop 512, the analysis of the spectrogram error measure) with a detailed view whose window, hop, frequency range, brightness, dynamic range, gamma, colormap and time zoom can be changed. All of this affects only playback and display; the files are not touched. A solution repository's site builder can copy the same script from the installed package (challenge_tools.build.player_js()) and should reference it with the cache-busting query player.js?v= followed by challenge_tools.build.player_version(), as the comparison pages do, so that a rebuilt site never serves a stale player from the browser cache. The playback.gain_db field of challenge.json is no longer used by the player and may be omitted.

Development#

pip install -e . pytest
pytest tests