# challenge-tools

Shared tooling for the instrument simulation challenges: the submission schema, validation of submissions, the error measures, the reference-selection rule, and the generator of the static comparison site. Every challenge repository (for example [challenges/string](https://vault1.magland.org/challenges/string) and [challenges/plate](https://vault1.magland.org/challenges/plate)) installs this package in its workflow and runs it against the solution repositories listed in its registry. Solution repositories run only the validator.

## Install

```
pip install git+https://vault1.magland.org/challenges/challenge-tools
```

## Commands

```
challenge-tools fetch    --registry registry/solution-repos.json --clones clones
challenge-tools validate --challenges challenges --registry registry/solution-repos.json --clones clones
challenge-tools build    --challenges challenges --registry registry/solution-repos.json --clones clones --out site
challenge-tools validate --challenges <spec-repo>/challenges --solutions solutions      # a solution repository checking itself
```

`fetch` clones or updates every registered repository (shallow, one branch, anonymously; `git lfs pull` is run when git-lfs is available). `validate` checks each submission JSON against [schema.json](schema.json), the wav files against the challenge's output specification, and any field exports against the challenge's recommended grid. `build` validates, computes every pairwise error, chooses the reference, and writes the site.

## What a challenge must provide

A directory `challenges/<challenge-id>/` with a `README.md` (the specification) and a `challenge.json` with at least:

```json
{
  "challenge_id": "...",
  "output": {"sample_rate_hz": 48000, "num_samples": 96000, "channels": 1, "sample_format": "float32", "signal_scale": 1.0},
  "metrics": {"primary": "waveform_l2", "waveform_l2": {}, "spectrogram_l2": {"n_fft": 2048, "hop": 512}},
  "fields": {"format": "hdf5", "axes": ["t", "x"], "shape": [4001, 201]},
  "reference_solution": null
}
```

`fields` is optional and describes the grid of the optional field export: datasets named after `axes` plus a dataset `u` of the given shape. `reference_solution` overrides the automatic reference with a `repo/submission/solution` key.

## Scoring and reference selection

The error measures live in `tools/challenge_tools/metrics.py` and the reference rule in `tools/challenge_tools/reference.py`. The challenge READMEs are the specification; this code is intended to follow them exactly. The rule: within a submission, `more_accurate_than` asserts a partial order; candidates for reference are the maximal elements of each submission's order; a candidate is scored by the fraction of asserted pairs from other submissions whose order it reproduces under the primary error; ties are broken by the candidate's convergence gap (error to its nearest asserted-worse sibling) and then by mean error to the other candidates.

## Playback

The wav files are exact signals with a loud strike transient and a much quieter ring, so played raw they sound faint. The sites use `tools/challenge_tools/player.js`, a Web Audio player that applies a page-wide playback gain (default from `playback.gain_db` in challenge.json, adjustable) and a fast limiter so the transient does not clip. This affects only playback; a solution repository's site builder can copy the same file from the installed package (`challenge_tools.build.player_js()`).

## Development

```
pip install -e . pytest
pytest tests
```