challenge-tools#

Shared tooling for the instrument simulation challenges: the submission schema, validation of submissions, the error measures, the reference-selection rule, and the generator of the static comparison site. Every challenge repository (for example challenges/string and challenges/plate) installs this package in its workflow and runs it against the solution repositories listed in its registry. Solution repositories run only the validator.

Install#

pip install git+https://vault1.magland.org/challenges/challenge-tools

Commands#

challenge-tools fetch    --registry registry/solution-repos.json --clones clones
challenge-tools validate --challenges challenges --registry registry/solution-repos.json --clones clones
challenge-tools build    --challenges challenges --registry registry/solution-repos.json --clones clones --out site
challenge-tools validate --challenges <spec-repo>/challenges --solutions solutions      # a solution repository checking itself

fetch clones or updates every registered repository (shallow, one branch, anonymously; git lfs pull is run when git-lfs is available). validate checks each submission JSON against schema.json, the wav files against the challenge's output specification, and any field exports against the challenge's recommended grid. build validates, computes every pairwise error, chooses the reference, and writes the site.

What a challenge must provide#

A directory challenges/<challenge-id>/ with a README.md (the specification) and a challenge.json with at least:

{
  "challenge_id": "...",
  "output": {"sample_rate_hz": 48000, "num_samples": 96000, "channels": 1, "sample_format": "float32", "signal_scale": 1.0},
  "metrics": {"primary": "waveform_l2", "waveform_l2": {}, "spectrogram_l2": {"n_fft": 2048, "hop": 512}},
  "fields": {"format": "hdf5", "axes": ["t", "x"], "shape": [4001, 201]},
  "reference_solution": null
}

fields is optional and describes the grid of the optional field export: datasets named after axes plus a dataset u of the given shape. reference_solution overrides the automatic reference with a repo/submission/solution key.

Scoring and reference selection#

The error measures live in tools/challenge_tools/metrics.py and the reference rule in tools/challenge_tools/reference.py. The challenge READMEs are the specification; this code is intended to follow them exactly. The rule: within a submission, more_accurate_than asserts a partial order; candidates for reference are the maximal elements of each submission's order; a candidate is scored by the fraction of asserted pairs from other submissions whose order it reproduces under the primary error; ties are broken by the candidate's convergence gap (error to its nearest asserted-worse sibling) and then by mean error to the other candidates.

Playback#

The wav files are exact signals with a loud strike transient and a much quieter ring, so played raw they sound faint. The sites use tools/challenge_tools/player.js, a Web Audio player that applies a page-wide playback gain (default from playback.gain_db in challenge.json, adjustable) and a fast limiter so the transient does not clip. This affects only playback; a solution repository's site builder can copy the same file from the installed package (challenge_tools.build.player_js()).

Development#

pip install -e . pytest
pytest tests