Add string-pluck-v1 challenge, solution schema, and scoring tools
18 changed files+1023−0
.gitignoreadded+7−0View file
@@ -0,0 +1,7 @@
1+__pycache__/
2+*.egg-info/
3+build/
4+dist/
5+.pytest_cache/
6+.challenge-cache/
7+site/
.mochi/workflows/test.ymladded+14−0View file
@@ -0,0 +1,14 @@
1+name: Test tools
2+on:
3+ push:
4+ workflow_dispatch:
5+jobs:
6+ test:
7+ runs-on: ubuntu-latest
8+ steps:
9+ - uses: actions/checkout@v4
10+ - uses: actions/setup-python@v5
11+ with:
12+ python-version: '3.12'
13+ - run: pip install . pytest
14+ - run: pytest -q tests
README.mdadded+41−0View file
@@ -0,0 +1,41 @@
1+# Instrument simulation challenges: string (tier 0)
2+
3+This repository defines simulation challenges for a damped string. It is the first, and simplest, repository in a collection of challenges for physics-based simulation of musical instruments, in which each challenge is a fully specified continuum model together with a precisely defined output signal and error measures. Solutions are not stored here; they are submitted to the companion repository [challenges/string-solutions](https://vault1.magland.org/challenges/string-solutions), and the results table is published at [challenges/string-solutions/site](https://vault1.magland.org/challenges/string-solutions/site/).
4+
5+## Challenges
6+
7+| ID | Title | Deliverable |
8+| --- | --- | --- |
9+| [string-pluck-v1](challenges/string-pluck-v1/README.md) | Plucked damped string, pickup velocity | 2 s mono wav, 48 kHz, float32 |
10+
11+Each challenge lives in `challenges/<challenge-id>/` and consists of a `README.md`, which is the authoritative specification, and a `challenge.json`, which repeats the parameters, the output format, and the error measures in machine-readable form. A challenge is never changed once solutions exist for it; a corrected or modified model gets a new ID (for example, a `-v2` suffix).
12+
13+## How a challenge is specified
14+
15+The README states the governing equations, boundary and initial conditions, and all parameter values in SI units, and defines the output as a function of continuous time. It then states exactly how that function is sampled into the deliverable file. The aim is that the true solution is a well-defined function that does not depend on any discretization, so that methods of very different kinds (modal, finite difference, finite element, integral equation) can be compared on equal terms.
16+
17+## How solutions are submitted
18+
19+A submission is a JSON file in the solutions repository at `solutions/<challenge-id>/<submission-id>.json`, validated against [schema/solution.schema.json](schema/solution.schema.json). One file holds one or more solutions from one submitter for one challenge. Each solution names its output file (a path relative to the JSON file, or a URL), and may report a method name, discretization parameters, self-reported wall-clock time, hardware, a link to the source code, and a link to a visualization. The submission is made by pull request (or by direct push, for collaborators) to the solutions repository. See that repository's README for the details.
20+
21+## How solutions are scored
22+
23+There is no externally provided ground truth. Every solution is compared with a reference solution chosen from among the submissions, using the error measures defined in the challenge README. All measures are reported; one of them is designated primary and is used for ranking.
24+
25+The reference is chosen as follows. Within a submission, the submitter may assert that some solutions are more accurate than others through the field `more_accurate_than` (a convergence series at increasing resolution is the typical case). Only orderings within a single submission are trusted. The candidates for reference are the maximal elements of each submission's ordering. A candidate is scored by the number of asserted pairs from other submissions whose ordering is reproduced by the primary error to that candidate; the candidate reproducing the largest fraction wins, and ties are broken by the smallest mean error to the other candidates. When no orderings are asserted at all, this rule reduces to choosing the candidate nearest the others. A challenge may override the automatic choice by naming a solution in `reference_solution` in its `challenge.json`.
26+
27+The rule has a clear limitation: it is only as good as the asserted orderings. A submission whose sequence is not in fact monotone (for example, a scheme with two error terms of opposite sign that cancel at one resolution) makes a false assertion, and a candidate that happens to reproduce that false pair can be preferred over a more accurate one. With only two submissions this cannot be detected automatically. The results page therefore shows, for every candidate, which asserted pairs it fails to reproduce, so that a reader can judge the choice, and the maintainers can override it with `reference_solution`. More independent submissions make the rule more reliable.
28+
29+The results page lists every solution with its errors relative to the reference, its reported time and hardware, an audio player, and links to code and visualizations. It also reports which asserted orderings the chosen reference does or does not reproduce, and the full matrix of pairwise errors.
30+
31+## Tools
32+
33+The `tools/` directory holds a small Python package, `challenge_tools`, that validates submissions, computes the error measures, selects the reference, and writes the static results site. The solutions repository runs it in a workflow on every push.
34+
35+```
36+pip install .
37+challenge-tools validate --challenges challenges --solutions ../string-solutions/solutions
38+challenge-tools build --challenges challenges --solutions ../string-solutions/solutions --out site
39+```
40+
41+The error measures are implemented in `tools/challenge_tools/metrics.py` and the reference rule in `tools/challenge_tools/reference.py`; the challenge README is the specification and the code is intended to follow it exactly.
challenges/string-pluck-v1/README.mdadded+121−0View file
@@ -0,0 +1,121 @@
1+# string-pluck-v1: plucked damped string, pickup velocity
2+
3+This is the hello-world challenge of the collection. The model is a linear damped string with fixed ends, released from rest from a smooth plucked shape. The deliverable is a two second mono wav file containing the transverse velocity of the string at a fixed pickup point. Every quantity below is defined in the continuum, so the true solution is a specific function of time that does not depend on any discretization. A modal series solution is given in the appendix; it converges to the true solution and can be evaluated to machine precision, so this challenge is mainly a test of the submission and scoring pipeline.
4+
5+The machine-readable version of this specification is [challenge.json](challenge.json).
6+
7+## Model
8+
9+Let $u(x, t)$ denote the transverse displacement in meters of the string at position $x \in [0, L]$ and time $t \ge 0$. The string satisfies
10+
11+$$
12+\frac{\partial^2 u}{\partial t^2} = c^2 \frac{\partial^2 u}{\partial x^2} - 2\sigma_0 \frac{\partial u}{\partial t} + \kappa \frac{\partial^3 u}{\partial x^2 \partial t}, \qquad 0 < x < L, \quad t > 0,
13+$$
14+
15+with fixed ends
16+
17+$$
18+u(0, t) = u(L, t) = 0, \qquad t \ge 0,
19+$$
20+
21+and initial conditions
22+
23+$$
24+u(x, 0) = u_0(x), \qquad \frac{\partial u}{\partial t}(x, 0) = 0, \qquad 0 \le x \le L.
25+$$
26+
27+Here $c$ is the wave speed, $\sigma_0$ is a frequency-independent loss rate, and $\kappa$ is a Kelvin-Voigt viscosity that damps high frequencies faster than low ones. The initial shape is a raised cosine bump of height $h$ and half-width $w$ centered at the pluck point $x_p$:
28+
29+$$
30+u_0(x) = \begin{cases} \dfrac{h}{2}\left(1 + \cos\dfrac{\pi (x - x_p)}{w}\right), & |x - x_p| \le w, \\[1ex] 0, & \text{otherwise.} \end{cases}
31+$$
32+
33+Note that $u_0$ is continuously differentiable and vanishes at both ends, since $w < x_p < L - w$.
34+
35+## Parameters
36+
37+| Symbol | Value | Unit | Meaning |
38+| --- | --- | --- | --- |
39+| $L$ | 0.65 | m | string length |
40+| $c$ | 143 | m/s | wave speed (fundamental $c / 2L = 110$ Hz) |
41+| $\sigma_0$ | 3 | 1/s | frequency-independent loss rate |
42+| $\kappa$ | 0.004 | m^2/s | Kelvin-Voigt viscosity |
43+| $h$ | 0.002 | m | pluck height |
44+| $x_p$ | 0.1625 | m | pluck center ($L/4$) |
45+| $w$ | 0.03 | m | pluck half-width |
46+| $x_m$ | 0.54 | m | pickup position |
47+
48+All values are exact decimal numbers, not rounded measurements.
49+
50+## Output signal
51+
52+The audio signal is the transverse velocity of the string at the pickup point,
53+
54+$$
55+s(t) = \frac{\partial u}{\partial t}(x_m, t), \qquad \text{in m/s}.
56+$$
57+
58+The deliverable is a wav file with the following properties.
59+
60+| Property | Value |
61+| --- | --- |
62+| channels | 1 |
63+| sample rate $f_s$ | 48000 Hz |
64+| number of samples $N$ | 96000 (2.0 s) |
65+| sample format | 32-bit IEEE float (WAVE_FORMAT_IEEE_FLOAT) |
66+| sample $k$, for $k = 0, \dots, N-1$ | $s(k / f_s) / s_\text{ref}$ |
67+| $s_\text{ref}$ | see `output.signal_scale` in challenge.json (m/s) |
68+
69+The first sample is at $t = 0$, where $s(0) = 0$. Samples are pointwise values of the continuum signal. No anti-aliasing filter is applied, even though $s(t)$ contains frequency content above the Nyquist frequency; that content is heavily damped by the Kelvin-Voigt term and is part of the exact answer. The scale $s_\text{ref}$ is a fixed constant chosen so that the peak sample magnitude is roughly 0.5. Do not normalize the signal; absolute amplitude is compared.
70+
71+## Error measures
72+
73+Let $a_k$ and $r_k$, $k = 0, \dots, N-1$, be the samples of a submitted solution and of the reference solution. Two error measures are computed. Both are relative and dimensionless, and both are reported for every solution.
74+
75+**Waveform error** (primary):
76+
77+$$
78+E_\text{wave} = \frac{\left( \sum_k (a_k - r_k)^2 \right)^{1/2}}{\left( \sum_k r_k^2 \right)^{1/2}}.
79+$$
80+
81+**Spectrogram error.** Let $M = 2048$ be the frame length and $H = 512$ the hop. The periodic Hann window is $g_j = \tfrac{1}{2}\left(1 - \cos(2\pi j / M)\right)$ for $j = 0, \dots, M-1$. Frame $m$ of a signal $x$ is $x_{mH + j}\, g_j$, for $m = 0, \dots, \lfloor (N - M) / H \rfloor$, with no zero padding at either end. Let $X_m[q]$, $q = 0, \dots, M/2$, be its discrete Fourier transform, $X_m[q] = \sum_{j=0}^{M-1} x_{mH+j}\, g_j\, e^{-2\pi i q j / M}$. Then
82+
83+$$
84+E_\text{spec} = \frac{\left( \sum_{m, q} \left( |A_m[q]| - |R_m[q]| \right)^2 \right)^{1/2}}{\left( \sum_{m, q} |R_m[q]|^2 \right)^{1/2}}.
85+$$
86+
87+The spectrogram error ignores phase within a frame and is less sensitive than the waveform error to small frequency errors that accumulate into phase drift over the two seconds.
88+
89+## Reference solution
90+
91+There is no externally supplied ground truth. The reference is chosen automatically from the submitted solutions by the procedure described in the [collection README](../../README.md), unless `reference_solution` in challenge.json names one explicitly. For this particular challenge the modal series in the appendix gives an essentially exact answer, which is expected to be selected.
92+
93+## Appendix: modal series solution
94+
95+This appendix is provided for convenience and verification; it is not part of the challenge definition, and a submission may use any method.
96+
97+Expanding $u(x, t) = \sum_{n \ge 1} a_n(t) \sin(k_n x)$ with $k_n = n\pi / L$, each coefficient satisfies
98+
99+$$
100+a_n'' + 2\alpha_n a_n' + c^2 k_n^2 a_n = 0, \qquad \alpha_n = \sigma_0 + \tfrac{1}{2}\kappa k_n^2,
101+$$
102+
103+with $a_n(0) = A_n$ and $a_n'(0) = 0$, where
104+
105+$$
106+A_n = \frac{2}{L} \int_0^L u_0(x) \sin(k_n x)\, dx = \frac{2h}{L}\, \sin(k_n x_p)\, \sin(k_n w)\, \frac{(\pi / w)^2}{k_n \left( (\pi/w)^2 - k_n^2 \right)}.
107+$$
108+
109+With the parameters above, $(\pi/w)^2 - k_n^2 \ne 0$ for every integer $n$, since $L / w$ is not an integer. For every mode that matters $\alpha_n < c k_n$ (the modes are underdamped up to roughly $n = 14800$), and in that case, with $\omega_n = \sqrt{c^2 k_n^2 - \alpha_n^2}$,
110+
111+$$
112+a_n(t) = A_n e^{-\alpha_n t} \left( \cos \omega_n t + \frac{\alpha_n}{\omega_n} \sin \omega_n t \right), \qquad a_n'(t) = -A_n \frac{c^2 k_n^2}{\omega_n} e^{-\alpha_n t} \sin \omega_n t.
113+$$
114+
115+The pickup velocity is then
116+
117+$$
118+s(t) = \sum_{n \ge 1} a_n'(t) \sin(k_n x_m).
119+$$
120+
121+Since $|A_n| = O(n^{-3})$ and $|a_n'|$ carries an extra factor of $n$, the terms decay like $n^{-2}$ before damping is taken into account, and for $t > 0$ the exponential factor makes the series converge rapidly. In practice a few hundred modes give the signal to single precision for all $t \ge 1 / f_s$; at $t = 0$ the exact value is $s(0) = 0$.
challenges/string-pluck-v1/challenge.jsonadded+35−0View file
@@ -0,0 +1,35 @@
1+{
2+ "challenge_id": "string-pluck-v1",
3+ "title": "Plucked damped string: pickup velocity",
4+ "version": 1,
5+ "output": {
6+ "type": "wav",
7+ "sample_rate_hz": 48000,
8+ "num_samples": 96000,
9+ "channels": 1,
10+ "sample_format": "float32",
11+ "signal": "transverse velocity at the pickup point",
12+ "signal_unit": "m/s",
13+ "signal_scale": 14.0
14+ },
15+ "parameters": {
16+ "L": 0.65,
17+ "c": 143.0,
18+ "sigma0": 3.0,
19+ "kappa": 0.004,
20+ "h": 0.002,
21+ "x_p": 0.1625,
22+ "w": 0.03,
23+ "x_m": 0.54
24+ },
25+ "metrics": {
26+ "primary": "waveform_l2",
27+ "waveform_l2": {},
28+ "spectrogram_l2": {
29+ "window": "hann_periodic",
30+ "n_fft": 2048,
31+ "hop": 512
32+ }
33+ },
34+ "reference_solution": null
35+}
pyproject.tomladded+20−0View file
@@ -0,0 +1,20 @@
1+[build-system]
2+requires = ["setuptools>=68"]
3+build-backend = "setuptools.build_meta"
4+
5+[project]
6+name = "challenge-tools"
7+version = "0.1.0"
8+description = "Validation, scoring, and results-site generation for the instrument simulation challenges"
9+requires-python = ">=3.10"
10+dependencies = ["numpy", "scipy", "jsonschema"]
11+
12+[project.scripts]
13+challenge-tools = "challenge_tools.cli:main"
14+
15+[tool.setuptools]
16+package-dir = {"" = "tools"}
17+packages = ["challenge_tools"]
18+
19+[tool.setuptools.package-data]
20+challenge_tools = ["*.json"]
schema/solution.schema.jsonadded+68−0View file
@@ -0,0 +1,68 @@
1+{
2+ "$schema": "https://json-schema.org/draft/2020-12/schema",
3+ "$id": "https://vault1.magland.org/challenges/string/schema/solution.schema.json",
4+ "title": "Challenge solution submission",
5+ "description": "One submission file: a set of solutions to one challenge from one submitter, with an optional accuracy partial order among them.",
6+ "type": "object",
7+ "required": ["schema_version", "submission_id", "challenge_id", "submitter", "solutions"],
8+ "additionalProperties": false,
9+ "properties": {
10+ "schema_version": { "const": 1 },
11+ "submission_id": {
12+ "type": "string",
13+ "pattern": "^[a-z0-9][a-z0-9-]*$",
14+ "description": "Identifier of this submission. Must equal the file name without .json."
15+ },
16+ "challenge_id": {
17+ "type": "string",
18+ "pattern": "^[a-z0-9][a-z0-9-]*$",
19+ "description": "The challenge this submission solves. Must equal the name of the directory containing the file."
20+ },
21+ "submitter": {
22+ "type": "object",
23+ "required": ["name"],
24+ "additionalProperties": false,
25+ "properties": {
26+ "name": { "type": "string" },
27+ "email": { "type": "string" },
28+ "url": { "type": "string", "format": "uri" }
29+ }
30+ },
31+ "date": { "type": "string", "format": "date" },
32+ "description": { "type": "string" },
33+ "code_url": { "type": "string", "format": "uri", "description": "Source code for the method, applying to every solution unless overridden." },
34+ "viz_url": { "type": "string", "format": "uri", "description": "A web application visualizing the solutions." },
35+ "solutions": {
36+ "type": "array",
37+ "minItems": 1,
38+ "items": {
39+ "type": "object",
40+ "required": ["solution_id", "output"],
41+ "additionalProperties": false,
42+ "properties": {
43+ "solution_id": {
44+ "type": "string",
45+ "pattern": "^[a-z0-9][a-z0-9-]*$",
46+ "description": "Unique within the submission."
47+ },
48+ "output": {
49+ "type": "string",
50+ "description": "Path to the wav file relative to the directory containing this JSON file, or an http(s) URL."
51+ },
52+ "method": { "type": "string", "description": "Short name of the numerical method." },
53+ "parameters": { "type": "object", "description": "Discretization parameters, free form." },
54+ "timing_seconds": { "type": "number", "minimum": 0, "description": "Self-reported wall-clock time to produce the output, excluding file I/O." },
55+ "hardware": { "type": "string" },
56+ "code_url": { "type": "string", "format": "uri" },
57+ "viz_url": { "type": "string", "format": "uri" },
58+ "more_accurate_than": {
59+ "type": "array",
60+ "items": { "type": "string" },
61+ "description": "solution_ids in this same file that the submitter asserts are less accurate than this one."
62+ },
63+ "notes": { "type": "string" }
64+ }
65+ }
66+ }
67+ }
68+}
tests/test_tools.pyadded+56−0View file
@@ -0,0 +1,56 @@
1+import numpy as np
2+
3+from challenge_tools.metrics import spectrogram, spectrogram_l2, waveform_l2
4+from challenge_tools.reference import OrderInfo, collect_order, maximal_elements, select_reference
5+
6+
7+def test_waveform_l2():
8+ r = np.array([1.0, 2.0, 2.0])
9+ assert waveform_l2(r, r) == 0.0
10+ assert abs(waveform_l2(r * 1.1, r) - 0.1) < 1e-12
11+
12+
13+def test_spectrogram_shape():
14+ x = np.zeros(96000)
15+ s = spectrogram(x, 2048, 512)
16+ assert s.shape == ((96000 - 2048) // 512 + 1, 1025)
17+
18+
19+def test_spectrogram_phase_insensitive():
20+ t = np.arange(96000) / 48000
21+ a = np.sin(2 * np.pi * 440 * t)
22+ b = np.sin(2 * np.pi * 440 * t + 1.0)
23+ assert waveform_l2(a, b) > 0.5
24+ assert spectrogram_l2(a, b, 2048, 512) < 1e-2
25+
26+
27+def test_reference_selection_prefers_order_preserving_candidate():
28+ subs = [
29+ {"submission_id": "x", "solutions": [
30+ {"solution_id": "a", "more_accurate_than": []},
31+ {"solution_id": "b", "more_accurate_than": ["a"]},
32+ ]},
33+ {"submission_id": "y", "solutions": [
34+ {"solution_id": "c", "more_accurate_than": []},
35+ {"solution_id": "d", "more_accurate_than": ["c"]},
36+ ]},
37+ ]
38+ info = collect_order(subs)
39+ assert set(maximal_elements(info)) == {"x/b", "y/d"}
40+ # Truth: x/b is exact; y/d is wrong. Errors to x/b respect y's chain, errors to y/d do not respect x's chain.
41+ keys = ["x/a", "x/b", "y/c", "y/d"]
42+ truth = {"x/a": 1.0, "x/b": 0.0, "y/c": 3.0, "y/d": 2.5}
43+ err = {a: {b: abs(truth[a] - truth[b]) for b in keys} for a in keys}
44+ # under y/d: err(x/b)=2.5 > err(x/a)=1.5, so x's pair is violated
45+ rep = select_reference(info, err)
46+ assert rep["reference"] == "x/b"
47+ assert rep["candidates"][0]["score"] == 1.0
48+
49+
50+def test_reference_selection_medoid_fallback():
51+ info = OrderInfo(pairs=[], submission_of={"p/a": "p", "q/b": "q", "r/c": "r"})
52+ keys = list(info.submission_of)
53+ pos = {"p/a": 0.0, "q/b": 1.0, "r/c": 5.0}
54+ err = {a: {b: abs(pos[a] - pos[b]) for b in keys} for a in keys}
55+ rep = select_reference(info, err)
56+ assert rep["reference"] == "q/b"
tools/challenge_tools/__init__.pyadded+1−0View file
@@ -0,0 +1 @@
1+"""Tools for the instrument simulation challenges: validation, metrics, reference selection, results site."""
tools/challenge_tools/__main__.pyadded+3−0View file
@@ -0,0 +1,3 @@
1+from .cli import main
2+
3+main()
tools/challenge_tools/build.pyadded+248−0View file
@@ -0,0 +1,248 @@
1+"""Compute all scores and write the results site."""
2+from __future__ import annotations
3+
4+import datetime as dt
5+import html
6+import json
7+import shutil
8+from pathlib import Path
9+
10+import numpy as np
11+
12+from .loading import load_challenges, load_submissions
13+from .metrics import compute_metric, metric_names
14+from .reference import collect_order, select_reference
15+from .wav import read_wav, resolve_output
16+
17+
18+def score_challenge(challenge: dict, submissions: list[dict], out_audio: Path | None = None) -> dict:
19+ """Load every output for one challenge, compute pairwise errors, choose the reference, and return a results dict."""
20+ spec = challenge["output"]
21+ names = metric_names(challenge)
22+ primary = challenge["metrics"]["primary"]
23+
24+ signals: dict[str, np.ndarray] = {}
25+ rows: dict[str, dict] = {}
26+ for sub in submissions:
27+ for sol in sub["solutions"]:
28+ key = f"{sub['submission_id']}/{sol['solution_id']}"
29+ p = resolve_output(sol["output"], sub["_path"].parent)
30+ signals[key] = read_wav(p, spec)
31+ audio_rel = None
32+ if out_audio is not None:
33+ dest = out_audio / sub["submission_id"] / f"{sol['solution_id']}.wav"
34+ dest.parent.mkdir(parents=True, exist_ok=True)
35+ shutil.copyfile(p, dest)
36+ audio_rel = f"audio/{sub['submission_id']}/{sol['solution_id']}.wav"
37+ rows[key] = {
38+ "key": key,
39+ "submission_id": sub["submission_id"],
40+ "solution_id": sol["solution_id"],
41+ "submitter": sub["submitter"]["name"],
42+ "date": sub.get("date"),
43+ "method": sol.get("method"),
44+ "parameters": sol.get("parameters"),
45+ "timing_seconds": sol.get("timing_seconds"),
46+ "hardware": sol.get("hardware"),
47+ "code_url": sol.get("code_url") or sub.get("code_url"),
48+ "viz_url": sol.get("viz_url") or sub.get("viz_url"),
49+ "more_accurate_than": [f"{sub['submission_id']}/{w}" for w in sol.get("more_accurate_than", [])],
50+ "notes": sol.get("notes"),
51+ "audio": audio_rel,
52+ "peak": float(np.max(np.abs(signals[key]))),
53+ }
54+
55+ keys = list(signals)
56+ pairwise = {name: {a: {} for a in keys} for name in names}
57+ for name in names:
58+ params = challenge["metrics"][name]
59+ for a in keys:
60+ for r in keys:
61+ pairwise[name][a][r] = 0.0 if a == r else compute_metric(name, signals[a], signals[r], params)
62+
63+ info = collect_order(submissions)
64+ explicit = challenge.get("reference_solution")
65+ if explicit is not None and explicit not in keys:
66+ raise KeyError(f"reference_solution {explicit!r} is not among the solutions")
67+ selection = select_reference(info, pairwise[primary], explicit)
68+ ref = selection["reference"]
69+ for key, row in rows.items():
70+ row["errors"] = {name: pairwise[name][key][ref] for name in names} if ref else {}
71+ row["is_reference"] = key == ref
72+
73+ # Check every asserted pair against the chosen reference, for the report.
74+ violations = []
75+ if ref:
76+ for worse, better, _sid in info.pairs:
77+ for name in names:
78+ if not pairwise[name][better][ref] < pairwise[name][worse][ref]:
79+ violations.append({"metric": name, "worse": worse, "better": better})
80+
81+ ordered = sorted(rows.values(), key=lambda r: (r["errors"].get(primary, float("inf")), r["key"]))
82+ return {
83+ "challenge_id": challenge["challenge_id"],
84+ "title": challenge.get("title"),
85+ "generated": dt.datetime.now(dt.timezone.utc).isoformat(timespec="seconds"),
86+ "metrics": names,
87+ "primary_metric": primary,
88+ "reference": ref,
89+ "reference_selection": selection,
90+ "order_violations_vs_reference": violations,
91+ "solutions": ordered,
92+ "pairwise": pairwise,
93+ "submissions": [
94+ {k: v for k, v in sub.items() if not k.startswith("_") and k != "solutions"} for sub in submissions
95+ ],
96+ }
97+
98+
99+# ---------------------------------------------------------------- HTML
100+
101+_CSS = """
102+:root { --fg: #1b1b1b; --bg: #ffffff; --muted: #666; --line: #ddd; --accent: #2a6db0; --ref: #e8f4e8; }
103+@media (prefers-color-scheme: dark) { :root { --fg: #e6e6e6; --bg: #151515; --muted: #999; --line: #333; --accent: #7ab3e8; --ref: #1f3320; } }
104+body { font-family: system-ui, sans-serif; color: var(--fg); background: var(--bg); margin: 0; padding: 16px; line-height: 1.45; }
105+main { max-width: 1200px; margin: 0 auto; }
106+a { color: var(--accent); }
107+table { border-collapse: collapse; width: 100%; font-size: 0.92em; }
108+th, td { border-bottom: 1px solid var(--line); padding: 6px 8px; text-align: left; vertical-align: top; }
109+th { position: sticky; top: 0; background: var(--bg); }
110+td.num { text-align: right; font-variant-numeric: tabular-nums; white-space: nowrap; }
111+tr.ref { background: var(--ref); }
112+.muted { color: var(--muted); }
113+.wrap { overflow-x: auto; }
114+audio { width: 180px; height: 28px; }
115+details { margin: 1em 0; }
116+code { font-size: 0.9em; }
117+"""
118+
119+
120+def _fmt(x, digits=4):
121+ if x is None:
122+ return ""
123+ if isinstance(x, float):
124+ if x == 0:
125+ return "0"
126+ return f"{x:.{digits}g}"
127+ return html.escape(str(x))
128+
129+
130+def _link(url, text):
131+ if not url:
132+ return ""
133+ return f'<a href="{html.escape(url)}">{html.escape(text)}</a>'
134+
135+
136+def render_challenge_page(res: dict, challenge: dict, links: dict) -> str:
137+ names = res["metrics"]
138+ primary = res["primary_metric"]
139+ head = "".join(
140+ f"<th>{html.escape(n)}{' (primary)' if n == primary else ''}</th>" for n in names
141+ )
142+ body = []
143+ for r in res["solutions"]:
144+ errs = "".join(f'<td class="num">{_fmt(r["errors"].get(n))}</td>' for n in names)
145+ params = html.escape(json.dumps(r["parameters"])) if r["parameters"] else ""
146+ mat = ", ".join(html.escape(k.split("/", 1)[1]) for k in r["more_accurate_than"])
147+ audio = f'<audio controls preload="none" src="{html.escape(r["audio"])}"></audio>' if r["audio"] else ""
148+ linkbits = " ".join(x for x in [_link(r["code_url"], "code"), _link(r["viz_url"], "viz"), _link(r["audio"], "wav")] if x)
149+ body.append(
150+ f'<tr class="{"ref" if r["is_reference"] else ""}">'
151+ f'<td>{html.escape(r["submission_id"])}<br><span class="muted">{html.escape(r["submitter"])}</span></td>'
152+ f'<td>{html.escape(r["solution_id"])}{" <b>(reference)</b>" if r["is_reference"] else ""}</td>'
153+ f'<td>{html.escape(r["method"] or "")}<br><span class="muted"><code>{params}</code></span></td>'
154+ f"{errs}"
155+ f'<td class="num">{_fmt(r["timing_seconds"], 3)}</td>'
156+ f'<td>{html.escape(r["hardware"] or "")}</td>'
157+ f'<td class="muted">{mat}</td>'
158+ f"<td>{audio}<br>{linkbits}</td>"
159+ "</tr>"
160+ )
161+ sel = res["reference_selection"]
162+ cand_rows = "".join(
163+ f"<tr><td>{html.escape(c['candidate'])}</td><td class='num'>{c['pairs_preserved']}/{c['pairs_total']}</td>"
164+ f"<td class='num'>{_fmt(c['mean_error_to_other_candidates'])}</td>"
165+ f"<td>{html.escape(', '.join(f'{w} vs {b}' for w, b in c['violated_pairs']))}</td></tr>"
166+ for c in sel.get("candidates", [])
167+ )
168+ viol = res["order_violations_vs_reference"]
169+ viol_html = (
170+ "<p>All asserted accuracy orderings are preserved by both metrics under the chosen reference.</p>"
171+ if not viol
172+ else "<ul>" + "".join(
173+ f"<li>{html.escape(v['metric'])}: {html.escape(v['better'])} asserted more accurate than {html.escape(v['worse'])}, but its error is not smaller</li>"
174+ for v in viol
175+ ) + "</ul>"
176+ )
177+ pair_tables = []
178+ keys = [r["key"] for r in res["solutions"]]
179+ for n in names:
180+ hdr = "".join(f"<th>{html.escape(k)}</th>" for k in keys)
181+ rws = "".join(
182+ f"<tr><th>{html.escape(a)}</th>" + "".join(f"<td class='num'>{_fmt(res['pairwise'][n][a][b], 3)}</td>" for b in keys) + "</tr>"
183+ for a in keys
184+ )
185+ pair_tables.append(f"<h4>{html.escape(n)}</h4><div class='wrap'><table><tr><th></th>{hdr}</tr>{rws}</table></div>")
186+ spec_link = links.get("spec")
187+ return f"""<!doctype html>
188+<html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1">
189+<title>{html.escape(res["challenge_id"])} results</title><style>{_CSS}</style></head>
190+<body><main>
191+<p><a href="../">All challenges</a></p>
192+<h1>{html.escape(res["challenge_id"])}</h1>
193+<p>{html.escape(res.get("title") or "")}. {"<a href='" + html.escape(spec_link) + "'>Challenge specification</a>. " if spec_link else ""}<a href="results.json">results.json</a>. Generated {html.escape(res["generated"])}.</p>
194+<p>Errors are relative to the reference solution <b>{html.escape(res["reference"] or "none")}</b>, chosen by rule <i>{html.escape(sel["rule"])}</i>. Rows are sorted by the primary metric.</p>
195+<div class="wrap"><table>
196+<tr><th>Submission</th><th>Solution</th><th>Method</th>{head}<th>Time (s)</th><th>Hardware</th><th>Claimed more accurate than</th><th>Output</th></tr>
197+{"".join(body)}
198+</table></div>
199+<details><summary>Reference selection</summary>
200+<p>Candidates are the maximal elements of each submission's asserted order. The score is the number of asserted pairs from other submissions whose order is preserved by the primary error to this candidate. Ties are broken by the mean primary error to the other candidates.</p>
201+<div class="wrap"><table><tr><th>Candidate</th><th>Pairs preserved</th><th>Mean error to other candidates</th><th>Violated pairs</th></tr>{cand_rows}</table></div>
202+<h4>Order check against the chosen reference</h4>{viol_html}
203+</details>
204+<details><summary>Pairwise errors (row vs column as reference)</summary>{"".join(pair_tables)}</details>
205+</main></body></html>
206+"""
207+
208+
209+def render_index(results: list[dict], links: dict) -> str:
210+ items = "".join(
211+ f'<li><a href="{html.escape(r["challenge_id"])}/">{html.escape(r["challenge_id"])}</a>: {html.escape(r.get("title") or "")} '
212+ f'<span class="muted">({len(r["solutions"])} solutions, reference {html.escape(r["reference"] or "none")})</span></li>'
213+ for r in results
214+ )
215+ repo = links.get("challenge_repo")
216+ sols = links.get("solutions_repo")
217+ return f"""<!doctype html>
218+<html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1">
219+<title>Challenge results</title><style>{_CSS}</style></head>
220+<body><main>
221+<h1>Challenge results</h1>
222+<p>{"Specifications: <a href='" + html.escape(repo) + "'>" + html.escape(repo) + "</a>. " if repo else ""}{"Solutions: <a href='" + html.escape(sols) + "'>" + html.escape(sols) + "</a>." if sols else ""}</p>
223+<ul>{items}</ul>
224+</main></body></html>
225+"""
226+
227+
228+def build(challenges_dir: Path, solutions_dir: Path, out: Path, challenge_repo_url: str | None = None, solutions_repo_url: str | None = None) -> list[dict]:
229+ challenges = load_challenges(challenges_dir)
230+ submissions = load_submissions(solutions_dir)
231+ out.mkdir(parents=True, exist_ok=True)
232+ results = []
233+ for cid, ch in challenges.items():
234+ subs = [s for s in submissions if s["challenge_id"] == cid]
235+ cdir = out / cid
236+ cdir.mkdir(parents=True, exist_ok=True)
237+ res = score_challenge(ch, subs, cdir / "audio") if subs else {
238+ "challenge_id": cid, "title": ch.get("title"), "generated": dt.datetime.now(dt.timezone.utc).isoformat(timespec="seconds"),
239+ "metrics": metric_names(ch), "primary_metric": ch["metrics"]["primary"], "reference": None,
240+ "reference_selection": {"rule": "none", "candidates": []}, "order_violations_vs_reference": [], "solutions": [], "pairwise": {}, "submissions": [],
241+ }
242+ with open(cdir / "results.json", "w") as f:
243+ json.dump(res, f, indent=1)
244+ links = {"spec": f"{challenge_repo_url}/blob/main/challenges/{cid}/README.md" if challenge_repo_url else None}
245+ (cdir / "index.html").write_text(render_challenge_page(res, ch, links))
246+ results.append(res)
247+ (out / "index.html").write_text(render_index(results, {"challenge_repo": challenge_repo_url, "solutions_repo": solutions_repo_url}))
248+ return results
tools/challenge_tools/cli.pyadded+53−0View file
@@ -0,0 +1,53 @@
1+from __future__ import annotations
2+
3+import argparse
4+import sys
5+from pathlib import Path
6+
7+
8+def main(argv=None):
9+ ap = argparse.ArgumentParser(prog="challenge-tools", description="Validate submissions and build the results site.")
10+ sp = ap.add_subparsers(dest="cmd", required=True)
11+
12+ v = sp.add_parser("validate", help="check submission files and their outputs")
13+ v.add_argument("--challenges", required=True, type=Path, help="directory containing <challenge-id>/challenge.json")
14+ v.add_argument("--solutions", required=True, type=Path, help="directory containing <challenge-id>/<submission-id>.json")
15+ v.add_argument("--no-outputs", action="store_true", help="skip reading the wav files")
16+
17+ b = sp.add_parser("build", help="compute scores and write the static results site")
18+ b.add_argument("--challenges", required=True, type=Path)
19+ b.add_argument("--solutions", required=True, type=Path)
20+ b.add_argument("--out", required=True, type=Path)
21+ b.add_argument("--challenge-repo-url", default=None)
22+ b.add_argument("--solutions-repo-url", default=None)
23+
24+ args = ap.parse_args(argv)
25+ if args.cmd == "validate":
26+ from .validate import validate
27+
28+ problems = validate(args.challenges, args.solutions, check_outputs=not args.no_outputs)
29+ for p in problems:
30+ print("PROBLEM:", p, file=sys.stderr)
31+ print(f"{len(problems)} problem(s)")
32+ return 1 if problems else 0
33+ if args.cmd == "build":
34+ from .validate import validate
35+ from .build import build
36+
37+ problems = validate(args.challenges, args.solutions)
38+ if problems:
39+ for p in problems:
40+ print("PROBLEM:", p, file=sys.stderr)
41+ return 1
42+ results = build(args.challenges, args.solutions, args.out, args.challenge_repo_url, args.solutions_repo_url)
43+ for r in results:
44+ print(f"{r['challenge_id']}: {len(r['solutions'])} solutions, reference = {r['reference']}")
45+ for s in r["solutions"]:
46+ errs = " ".join(f"{k}={v:.3e}" for k, v in s["errors"].items())
47+ print(f" {s['key']:40s} {errs} t={s['timing_seconds']}")
48+ return 0
49+ return 2
50+
51+
52+if __name__ == "__main__":
53+ sys.exit(main())
tools/challenge_tools/loading.pyadded+27−0View file
@@ -0,0 +1,27 @@
1+"""Loading challenge definitions and submission files."""
2+from __future__ import annotations
3+
4+import json
5+from pathlib import Path
6+
7+
8+def load_challenges(challenges_dir: Path) -> dict[str, dict]:
9+ out = {}
10+ for p in sorted(Path(challenges_dir).glob("*/challenge.json")):
11+ with open(p) as f:
12+ ch = json.load(f)
13+ ch["_dir"] = p.parent
14+ out[ch["challenge_id"]] = ch
15+ return out
16+
17+
18+def load_submissions(solutions_dir: Path) -> list[dict]:
19+ """Return every submission JSON under solutions/<challenge-id>/*.json, with _path and _dir_challenge set."""
20+ out = []
21+ for p in sorted(Path(solutions_dir).glob("*/*.json")):
22+ with open(p) as f:
23+ sub = json.load(f)
24+ sub["_path"] = p
25+ sub["_dir_challenge"] = p.parent.name
26+ out.append(sub)
27+ return out
tools/challenge_tools/metrics.pyadded+36−0View file
@@ -0,0 +1,36 @@
1+"""Error measures between a solution and a reference, as defined in each challenge README."""
2+from __future__ import annotations
3+
4+import numpy as np
5+
6+
7+def waveform_l2(a: np.ndarray, r: np.ndarray) -> float:
8+ return float(np.linalg.norm(a - r) / np.linalg.norm(r))
9+
10+
11+def spectrogram(x: np.ndarray, n_fft: int, hop: int) -> np.ndarray:
12+ """Magnitude STFT with a periodic Hann window, no padding. Shape (frames, n_fft//2 + 1)."""
13+ j = np.arange(n_fft)
14+ g = 0.5 * (1.0 - np.cos(2.0 * np.pi * j / n_fft))
15+ m = (len(x) - n_fft) // hop + 1
16+ idx = j[None, :] + hop * np.arange(m)[:, None]
17+ frames = x[idx] * g[None, :]
18+ return np.abs(np.fft.rfft(frames, axis=1))
19+
20+
21+def spectrogram_l2(a: np.ndarray, r: np.ndarray, n_fft: int, hop: int) -> float:
22+ sa = spectrogram(a, n_fft, hop)
23+ sr = spectrogram(r, n_fft, hop)
24+ return float(np.linalg.norm(sa - sr) / np.linalg.norm(sr))
25+
26+
27+def compute_metric(name: str, a: np.ndarray, r: np.ndarray, params: dict) -> float:
28+ if name == "waveform_l2":
29+ return waveform_l2(a, r)
30+ if name == "spectrogram_l2":
31+ return spectrogram_l2(a, r, int(params["n_fft"]), int(params["hop"]))
32+ raise KeyError(f"unknown metric {name}")
33+
34+
35+def metric_names(challenge: dict) -> list[str]:
36+ return [k for k in challenge["metrics"] if k != "primary"]
tools/challenge_tools/reference.pyadded+83−0View file
@@ -0,0 +1,83 @@
1+"""Choosing the reference solution from the submitted solutions.
2+
3+Each submission may assert a partial order among its own solutions through
4+``more_accurate_than``. Only orderings within a submission are trusted. A candidate
5+reference R is scored by the fraction of asserted pairs (worse, better) from *other*
6+submissions for which err(better, R) < err(worse, R) under the primary metric.
7+Candidates are the maximal elements of each submission's order. Ties are broken by the
8+smallest mean primary error to the other candidates; with no asserted pairs at all this
9+tie-break is the whole rule, so the reference is then the medoid of the candidates.
10+"""
11+from __future__ import annotations
12+
13+from dataclasses import dataclass, field
14+
15+
16+@dataclass
17+class OrderInfo:
18+ # (worse, better, submission_id) triples over global solution keys "submission/solution"
19+ pairs: list[tuple[str, str, str]] = field(default_factory=list)
20+ # global key -> submission id
21+ submission_of: dict[str, str] = field(default_factory=dict)
22+
23+
24+def collect_order(submissions: list[dict]) -> OrderInfo:
25+ info = OrderInfo()
26+ for sub in submissions:
27+ sid = sub["submission_id"]
28+ for sol in sub["solutions"]:
29+ key = f"{sid}/{sol['solution_id']}"
30+ info.submission_of[key] = sid
31+ for worse in sol.get("more_accurate_than", []):
32+ info.pairs.append((f"{sid}/{worse}", key, sid))
33+ return info
34+
35+
36+def maximal_elements(info: OrderInfo) -> list[str]:
37+ dominated = {worse for worse, _better, _sid in info.pairs}
38+ return [k for k in info.submission_of if k not in dominated]
39+
40+
41+def select_reference(
42+ info: OrderInfo,
43+ err: dict[str, dict[str, float]],
44+ explicit: str | None = None,
45+) -> dict:
46+ """Return a report dict with keys: reference, candidates (list of dicts), rule."""
47+ if explicit is not None:
48+ return {"reference": explicit, "rule": "explicit", "candidates": []}
49+
50+ candidates = maximal_elements(info)
51+ rows = []
52+ for cand in candidates:
53+ external = [p for p in info.pairs if p[2] != info.submission_of[cand]]
54+ preserved = 0
55+ violated = []
56+ for worse, better, _sid in external:
57+ if err[better][cand] < err[worse][cand]:
58+ preserved += 1
59+ else:
60+ violated.append([worse, better])
61+ others = [c for c in candidates if c != cand]
62+ mean_err = sum(err[c][cand] for c in others) / len(others) if others else 0.0
63+ rows.append(
64+ {
65+ "candidate": cand,
66+ "pairs_total": len(external),
67+ "pairs_preserved": preserved,
68+ "score": (preserved / len(external)) if external else None,
69+ "mean_error_to_other_candidates": mean_err,
70+ "violated_pairs": violated,
71+ }
72+ )
73+
74+ def sort_key(row):
75+ score = row["score"] if row["score"] is not None else -1.0
76+ return (-score, row["mean_error_to_other_candidates"])
77+
78+ rows.sort(key=sort_key)
79+ return {
80+ "reference": rows[0]["candidate"] if rows else None,
81+ "rule": "partial-order-agreement",
82+ "candidates": rows,
83+ }
tools/challenge_tools/solution.schema.jsonadded+68−0View file
@@ -0,0 +1,68 @@
1+{
2+ "$schema": "https://json-schema.org/draft/2020-12/schema",
3+ "$id": "https://vault1.magland.org/challenges/string/schema/solution.schema.json",
4+ "title": "Challenge solution submission",
5+ "description": "One submission file: a set of solutions to one challenge from one submitter, with an optional accuracy partial order among them.",
6+ "type": "object",
7+ "required": ["schema_version", "submission_id", "challenge_id", "submitter", "solutions"],
8+ "additionalProperties": false,
9+ "properties": {
10+ "schema_version": { "const": 1 },
11+ "submission_id": {
12+ "type": "string",
13+ "pattern": "^[a-z0-9][a-z0-9-]*$",
14+ "description": "Identifier of this submission. Must equal the file name without .json."
15+ },
16+ "challenge_id": {
17+ "type": "string",
18+ "pattern": "^[a-z0-9][a-z0-9-]*$",
19+ "description": "The challenge this submission solves. Must equal the name of the directory containing the file."
20+ },
21+ "submitter": {
22+ "type": "object",
23+ "required": ["name"],
24+ "additionalProperties": false,
25+ "properties": {
26+ "name": { "type": "string" },
27+ "email": { "type": "string" },
28+ "url": { "type": "string", "format": "uri" }
29+ }
30+ },
31+ "date": { "type": "string", "format": "date" },
32+ "description": { "type": "string" },
33+ "code_url": { "type": "string", "format": "uri", "description": "Source code for the method, applying to every solution unless overridden." },
34+ "viz_url": { "type": "string", "format": "uri", "description": "A web application visualizing the solutions." },
35+ "solutions": {
36+ "type": "array",
37+ "minItems": 1,
38+ "items": {
39+ "type": "object",
40+ "required": ["solution_id", "output"],
41+ "additionalProperties": false,
42+ "properties": {
43+ "solution_id": {
44+ "type": "string",
45+ "pattern": "^[a-z0-9][a-z0-9-]*$",
46+ "description": "Unique within the submission."
47+ },
48+ "output": {
49+ "type": "string",
50+ "description": "Path to the wav file relative to the directory containing this JSON file, or an http(s) URL."
51+ },
52+ "method": { "type": "string", "description": "Short name of the numerical method." },
53+ "parameters": { "type": "object", "description": "Discretization parameters, free form." },
54+ "timing_seconds": { "type": "number", "minimum": 0, "description": "Self-reported wall-clock time to produce the output, excluding file I/O." },
55+ "hardware": { "type": "string" },
56+ "code_url": { "type": "string", "format": "uri" },
57+ "viz_url": { "type": "string", "format": "uri" },
58+ "more_accurate_than": {
59+ "type": "array",
60+ "items": { "type": "string" },
61+ "description": "solution_ids in this same file that the submitter asserts are less accurate than this one."
62+ },
63+ "notes": { "type": "string" }
64+ }
65+ }
66+ }
67+ }
68+}
tools/challenge_tools/validate.pyadded+90−0View file
@@ -0,0 +1,90 @@
1+"""Validation of submission files against the schema and the challenge output spec."""
2+from __future__ import annotations
3+
4+import json
5+from importlib import resources
6+from pathlib import Path
7+
8+import jsonschema
9+
10+from .loading import load_challenges, load_submissions
11+from .wav import read_wav, resolve_output
12+
13+
14+def _schema() -> dict:
15+ with resources.files("challenge_tools").joinpath("solution.schema.json").open() as f:
16+ return json.load(f)
17+
18+
19+def _acyclic(edges: list[tuple[str, str]], nodes: list[str]) -> bool:
20+ adj = {n: [] for n in nodes}
21+ for a, b in edges:
22+ adj[a].append(b)
23+ state = {n: 0 for n in nodes}
24+
25+ def visit(n):
26+ state[n] = 1
27+ for m in adj[n]:
28+ if state[m] == 1 or (state[m] == 0 and not visit(m)):
29+ return False
30+ state[n] = 2
31+ return True
32+
33+ return all(state[n] == 2 or visit(n) for n in nodes)
34+
35+
36+def validate(challenges_dir: Path, solutions_dir: Path, check_outputs: bool = True) -> list[str]:
37+ """Return a list of problems (empty means valid)."""
38+ problems: list[str] = []
39+ challenges = load_challenges(challenges_dir)
40+ if not challenges:
41+ problems.append(f"no challenge.json found under {challenges_dir}")
42+ schema = _schema()
43+ validator = jsonschema.Draft202012Validator(schema, format_checker=jsonschema.FormatChecker())
44+ seen = set()
45+ for sub in load_submissions(solutions_dir):
46+ path: Path = sub["_path"]
47+ tag = str(path)
48+ errs = sorted(validator.iter_errors({k: v for k, v in sub.items() if not k.startswith("_")}), key=str)
49+ for e in errs:
50+ loc = "/".join(str(x) for x in e.absolute_path)
51+ problems.append(f"{tag}: schema: {loc}: {e.message}")
52+ if errs:
53+ continue
54+ if sub["submission_id"] != path.stem:
55+ problems.append(f"{tag}: submission_id {sub['submission_id']!r} does not match file name")
56+ if sub["challenge_id"] != sub["_dir_challenge"]:
57+ problems.append(f"{tag}: challenge_id {sub['challenge_id']!r} does not match directory {sub['_dir_challenge']!r}")
58+ if sub["challenge_id"] not in challenges:
59+ problems.append(f"{tag}: unknown challenge_id {sub['challenge_id']!r}")
60+ continue
61+ key = (sub["challenge_id"], sub["submission_id"])
62+ if key in seen:
63+ problems.append(f"{tag}: duplicate submission_id for this challenge")
64+ seen.add(key)
65+ ids = [s["solution_id"] for s in sub["solutions"]]
66+ if len(set(ids)) != len(ids):
67+ problems.append(f"{tag}: duplicate solution_id")
68+ edges = []
69+ for s in sub["solutions"]:
70+ for worse in s.get("more_accurate_than", []):
71+ if worse not in ids:
72+ problems.append(f"{tag}: {s['solution_id']}: more_accurate_than names unknown solution {worse!r}")
73+ elif worse == s["solution_id"]:
74+ problems.append(f"{tag}: {s['solution_id']}: more_accurate_than names itself")
75+ else:
76+ edges.append((worse, s["solution_id"]))
77+ if not _acyclic(edges, ids):
78+ problems.append(f"{tag}: more_accurate_than contains a cycle")
79+ if check_outputs:
80+ spec = challenges[sub["challenge_id"]]["output"]
81+ for s in sub["solutions"]:
82+ try:
83+ p = resolve_output(s["output"], path.parent)
84+ if not p.exists():
85+ problems.append(f"{tag}: {s['solution_id']}: output not found: {s['output']}")
86+ continue
87+ read_wav(p, spec)
88+ except Exception as e: # noqa: BLE001
89+ problems.append(f"{tag}: {s['solution_id']}: {e}")
90+ return problems
tools/challenge_tools/wav.pyadded+52−0View file
@@ -0,0 +1,52 @@
1+"""Reading and checking wav deliverables."""
2+from __future__ import annotations
3+
4+import hashlib
5+import os
6+import urllib.request
7+from pathlib import Path
8+
9+import numpy as np
10+from scipy.io import wavfile
11+
12+
13+def resolve_output(output: str, json_dir: Path, cache_dir: Path | None = None) -> Path:
14+ """Return a local path for a solution output, downloading URLs into cache_dir."""
15+ if output.startswith("http://") or output.startswith("https://"):
16+ cache_dir = cache_dir or Path(os.environ.get("CHALLENGE_CACHE", ".challenge-cache"))
17+ cache_dir.mkdir(parents=True, exist_ok=True)
18+ name = hashlib.sha256(output.encode()).hexdigest()[:16] + ".wav"
19+ dest = cache_dir / name
20+ if not dest.exists():
21+ urllib.request.urlretrieve(output, dest)
22+ return dest
23+ p = (json_dir / output).resolve()
24+ return p
25+
26+
27+def read_wav(path: Path, spec: dict) -> np.ndarray:
28+ """Read a wav and check it against the challenge's output spec. Returns float64 samples (mono)."""
29+ rate, data = wavfile.read(path)
30+ problems = []
31+ if rate != spec["sample_rate_hz"]:
32+ problems.append(f"sample rate {rate} != {spec['sample_rate_hz']}")
33+ if spec.get("sample_format") == "float32" and data.dtype != np.float32:
34+ problems.append(f"sample format {data.dtype} != float32")
35+ channels = 1 if data.ndim == 1 else data.shape[1]
36+ if channels != spec["channels"]:
37+ problems.append(f"channels {channels} != {spec['channels']}")
38+ n = data.shape[0]
39+ if n != spec["num_samples"]:
40+ problems.append(f"num samples {n} != {spec['num_samples']}")
41+ if not np.all(np.isfinite(data)):
42+ problems.append("non-finite samples")
43+ if problems:
44+ raise ValueError(f"{path}: " + "; ".join(problems))
45+ if data.ndim > 1:
46+ data = data[:, 0]
47+ return data.astype(np.float64)
48+
49+
50+def write_wav(path: Path, samples: np.ndarray, sample_rate: int) -> None:
51+ path.parent.mkdir(parents=True, exist_ok=True)
52+ wavfile.write(path, sample_rate, np.asarray(samples, dtype=np.float32))