dc14526Shared challenge tools: schema, validation, metrics, reference selection, comparison site; moved out of the string repositoryJeremy Magland 1# challenge-tools
3Shared tooling for the instrument simulation challenges: the submission schema, validation of submissions, the error measures, the reference-selection rule, and the generator of the static comparison site. Every challenge repository (for example [challenges/string](https://vault1.magland.org/challenges/string) and [challenges/plate](https://vault1.magland.org/challenges/plate)) installs this package in its workflow and runs it against the solution repositories listed in its registry. Solution repositories run only the validator.
5## Install
7```
8pip install git+https://vault1.magland.org/challenges/challenge-tools
9```
11## Commands
13```
14challenge-tools fetch --registry registry/solution-repos.json --clones clones
15challenge-tools validate --challenges challenges --registry registry/solution-repos.json --clones clones
16challenge-tools build --challenges challenges --registry registry/solution-repos.json --clones clones --out site
17challenge-tools validate --challenges <spec-repo>/challenges --solutions solutions # a solution repository checking itself
18```
20`fetch` clones or updates every registered repository (shallow, one branch, anonymously; `git lfs pull` is run when git-lfs is available). `validate` checks each submission JSON against [schema.json](schema.json), the wav files against the challenge's output specification, and any field exports against the challenge's recommended grid. `build` validates, computes every pairwise error, chooses the reference, and writes the site.
22## What a challenge must provide
24A directory `challenges/<challenge-id>/` with a `README.md` (the specification) and a `challenge.json` with at least:
26```json
27{
28 "challenge_id": "...",
29 "output": {"sample_rate_hz": 48000, "num_samples": 96000, "channels": 1, "sample_format": "float32", "signal_scale": 1.0},
30 "metrics": {"primary": "waveform_l2", "waveform_l2": {}, "spectrogram_l2": {"n_fft": 2048, "hop": 512}},
31 "fields": {"format": "hdf5", "axes": ["t", "x"], "shape": [4001, 201]},
32 "reference_solution": null
33}
34```
f99e1d0Interactive 3D geometry view: viewer.js (raw WebGL) renders a challenge's scene.json on its results pageJeremy Magland 36A challenge directory may also hold a `scene.json`, an interactive 3D view of the geometry that `build` copies to the site and embeds on the challenge page with `tools/challenge_tools/viewer.js` (raw WebGL, no dependencies; drag to rotate, wheel to zoom, right-drag to pan). Its format: `surfaces`, each with `vertices` (a list of `[x, y, z]`) and either `faces` (triangles as vertex indices) or `grid: [n1, n2]` for vertices listed in row-major order over a structured grid (`periodic: true` closes the second index), optionally with a per-vertex `scalar` (`name`, `values`) drawn with a colormap; `points`, each with `name`, `position`, optional `radius`, `color` and `far: true` for points outside the default framing such as a listener; `lines` with `from` and `to`; and `units` and `note` strings. The challenge repository generates the file from its own parameters; the tools do not know any geometry.
dc14526Shared challenge tools: schema, validation, metrics, reference selection, comparison site; moved out of the string repositoryJeremy Magland 38`fields` is optional and describes the grid of the optional field export: datasets named after `axes` plus a dataset `u` of the given shape. `reference_solution` overrides the automatic reference with a `repo/submission/solution` key.
40## Scoring and reference selection
42The error measures live in `tools/challenge_tools/metrics.py` and the reference rule in `tools/challenge_tools/reference.py`. The challenge READMEs are the specification; this code is intended to follow them exactly. The rule: within a submission, `more_accurate_than` asserts a partial order; candidates for reference are the maximal elements of each submission's order; a candidate is scored by the fraction of asserted pairs from other submissions whose order it reproduces under the primary error; ties are broken by the candidate's convergence gap (error to its nearest asserted-worse sibling) and then by mean error to the other candidates.
68a49aaPlayer: lower ceilings (limiter -10 dBFS, clip -6 dBFS), confirm before the first playback, cache-busted script tagJeremy Magland 46The wav files are exact signals with a loud strike transient and a much quieter ring, and they are at physical scale, so playing them raw is either faint or, with a gain, a hearing hazard on headphones. The sites use `tools/challenge_tools/player.js`, a Web Audio player built for safety first: each file is loudness-normalized at load time so that its loudest 400 ms window sits at -16 dBFS rms (about the loudness of a streamed music track), a fast limiter holds the strike transient at -10 dBFS, and a hard clipper at -6 dBFS is the last stage of the graph and cannot be bypassed, so nothing louder than a quiet music peak reaches the output whatever the settings. The first playback on a page asks for confirmation. The listener adjusts a bounded level offset (-20 to +6 dB, starting at -12 dB in a new browser and remembered there); raising it above the normalized level, or switching the limiter off, asks for confirmation. The player also draws a spectrogram of each file on request (Hann 2048, hop 512, the analysis of the spectrogram error measure) with a detailed view whose window, hop, frequency range, brightness, dynamic range, gamma, colormap and time zoom can be changed. All of this affects only playback and display; the files are not touched. A solution repository's site builder can copy the same script from the installed package (`challenge_tools.build.player_js()`) and should reference it with the cache-busting query `player.js?v=` followed by `challenge_tools.build.player_version()`, as the comparison pages do, so that a rebuilt site never serves a stale player from the browser cache. The `playback.gain_db` field of challenge.json is no longer used by the player and may be omitted.
dc14526Shared challenge tools: schema, validation, metrics, reference selection, comparison site; moved out of the string repositoryJeremy Magland 48## Development
50```
51pip install -e . pytest
52pytest tests
53```