1# challenge-tools
3Shared tooling for the instrument simulation challenges: the submission schema, validation of submissions, the error measures, the reference-selection rule, and the generator of the static comparison site. Every challenge repository (for example [challenges/string](https://vault1.magland.org/challenges/string) and [challenges/plate](https://vault1.magland.org/challenges/plate)) installs this package in its workflow and runs it against the solution repositories listed in its registry. Solution repositories run only the validator.
5## Install
7```
8pip install git+https://vault1.magland.org/challenges/challenge-tools
9```
11## Commands
13```
14challenge-tools fetch --registry registry/solution-repos.json --clones clones
15challenge-tools validate --challenges challenges --registry registry/solution-repos.json --clones clones
16challenge-tools build --challenges challenges --registry registry/solution-repos.json --clones clones --out site
17challenge-tools validate --challenges <spec-repo>/challenges --solutions solutions # a solution repository checking itself
18```
20`fetch` clones or updates every registered repository (shallow, one branch, anonymously; `git lfs pull` is run when git-lfs is available). `validate` checks each submission JSON against [schema.json](schema.json), the wav files against the challenge's output specification, and any field exports against the challenge's recommended grid. `build` validates, computes every pairwise error, chooses the reference, and writes the site.
22## What a challenge must provide
24A directory `challenges/<challenge-id>/` with a `README.md` (the specification) and a `challenge.json` with at least:
26```json
27{
28 "challenge_id": "...",
29 "output": {"sample_rate_hz": 48000, "num_samples": 96000, "channels": 1, "sample_format": "float32", "signal_scale": 1.0},
30 "metrics": {"primary": "waveform_l2", "waveform_l2": {}, "spectrogram_l2": {"n_fft": 2048, "hop": 512}},
31 "fields": {"format": "hdf5", "axes": ["t", "x"], "shape": [4001, 201]},
32 "reference_solution": null
33}
34```
36`fields` is optional and describes the grid of the optional field export: datasets named after `axes` plus a dataset `u` of the given shape. `reference_solution` overrides the automatic reference with a `repo/submission/solution` key.
38## Scoring and reference selection
40The error measures live in `tools/challenge_tools/metrics.py` and the reference rule in `tools/challenge_tools/reference.py`. The challenge READMEs are the specification; this code is intended to follow them exactly. The rule: within a submission, `more_accurate_than` asserts a partial order; candidates for reference are the maximal elements of each submission's order; a candidate is scored by the fraction of asserted pairs from other submissions whose order it reproduces under the primary error; ties are broken by the candidate's convergence gap (error to its nearest asserted-worse sibling) and then by mean error to the other candidates.
42## Playback
44The wav files are exact signals with a loud strike transient and a much quieter ring, and they are at physical scale, so playing them raw is either faint or, with a gain, a hearing hazard on headphones. The sites use `tools/challenge_tools/player.js`, a Web Audio player built for safety first: each file is loudness-normalized at load time so that its loudest 400 ms window sits at -16 dBFS rms (about the loudness of a streamed music track), a fast limiter tames the transient, and a hard clipper at -1 dBFS is the last stage of the graph and cannot be bypassed, so nothing louder than a normal music peak reaches the output whatever the settings. The listener adjusts a bounded level offset (-20 to +6 dB, starting at -12 dB in a new browser and remembered there); raising it above the normalized level, or switching the limiter off, asks for confirmation. The player also draws a spectrogram of each file on request (Hann 2048, hop 512, the analysis of the spectrogram error measure) with a detailed view whose window, hop, frequency range, brightness, dynamic range, gamma, colormap and time zoom can be changed. All of this affects only playback and display; the files are not touched. A solution repository's site builder can copy the same script from the installed package (`challenge_tools.build.player_js()`). The `playback.gain_db` field of challenge.json is no longer used by the player and may be omitted.
46## Development
48```
49pip install -e . pytest
50pytest tests
51```