Register solution repositories and build the comparison site here; optional HDF5 field export in the schema and spec
14 changed files+482−101
.gitignoremodified+1−0View file
@@ -5,3 +5,4 @@ dist/
55 .pytest_cache/
66 .challenge-cache/
77 site/
8+clones/
.mochi/workflows/results.ymladded+30−0View file
@@ -0,0 +1,30 @@
1+name: Results
2+on:
3+ push:
4+ workflow_dispatch:
5+concurrency:
6+ group: results
7+ cancel-in-progress: true
8+jobs:
9+ results:
10+ runs-on: ubuntu-latest
11+ steps:
12+ - uses: actions/checkout@v4
13+ - uses: actions/setup-python@v5
14+ with:
15+ python-version: '3.12'
16+ - run: pip install .
17+ - name: Fetch the registered solution repositories
18+ run: challenge-tools fetch --registry registry/solution-repos.json --clones clones --vault-url https://vault1.magland.org
19+ - name: Build the comparison site
20+ run: >
21+ challenge-tools build --challenges challenges --registry registry/solution-repos.json --clones clones
22+ --out site --vault-url https://vault1.magland.org --challenge-repo-url https://vault1.magland.org/challenges/string
23+ - if: github.ref == 'refs/heads/main'
24+ uses: actions/configure-pages@v4
25+ - if: github.ref == 'refs/heads/main'
26+ uses: actions/upload-pages-artifact@v3
27+ with:
28+ path: site
29+ - if: github.ref == 'refs/heads/main'
30+ uses: actions/deploy-pages@v4
README.mdmodified+12−9View file
@@ -1,6 +1,6 @@
11 # Instrument simulation challenges: string (tier 0)
22
3-This repository defines simulation challenges for a damped string. It is the first, and simplest, repository in a collection of challenges for physics-based simulation of musical instruments, in which each challenge is a fully specified continuum model together with a precisely defined output signal and error measures. Solutions are not stored here; they are submitted to the companion repository [challenges/string-solutions](https://vault1.magland.org/challenges/string-solutions), and the results table is published at [challenges/string-solutions/site](https://vault1.magland.org/challenges/string-solutions/site/).
3+This repository defines simulation challenges for a damped string, and publishes the comparison of every registered solution. It is the first, and simplest, repository in a collection of challenges for physics-based simulation of musical instruments, in which each challenge is a fully specified continuum model together with a precisely defined output signal and error measures. Solutions live in separate repositories that are registered here; the comparison is at [challenges/string/site](https://vault1.magland.org/challenges/string/site/).
44
55 ## Challenges
66
@@ -8,15 +8,17 @@ This repository defines simulation challenges for a damped string. It is the fir
88 | --- | --- | --- |
99 | [string-pluck-v1](challenges/string-pluck-v1/README.md) | Plucked damped string, pickup velocity | 2 s mono wav, 48 kHz, float32 |
1010
11-Each challenge lives in `challenges/<challenge-id>/` and consists of a `README.md`, which is the authoritative specification, and a `challenge.json`, which repeats the parameters, the output format, and the error measures in machine-readable form. A challenge is never changed once solutions exist for it; a corrected or modified model gets a new ID (for example, a `-v2` suffix).
11+Each challenge lives in `challenges/<challenge-id>/` and consists of a `README.md`, which is the authoritative specification, and a `challenge.json`, which repeats the parameters, the output format, the error measures, and the recommended grid for optional field exports in machine-readable form. A challenge is never changed once solutions exist for it; a corrected or modified model gets a new ID (for example, a `-v2` suffix).
1212
1313 ## How a challenge is specified
1414
1515 The README states the governing equations, boundary and initial conditions, and all parameter values in SI units, and defines the output as a function of continuous time. It then states exactly how that function is sampled into the deliverable file. The aim is that the true solution is a well-defined function that does not depend on any discretization, so that methods of very different kinds (modal, finite difference, finite element, integral equation) can be compared on equal terms.
1616
17-## How solutions are submitted
17+## Solution repositories and the registry
1818
19-A submission is a JSON file in the solutions repository at `solutions/<challenge-id>/<submission-id>.json`, validated against [schema/solution.schema.json](schema/solution.schema.json). One file holds one or more solutions from one submitter for one challenge. Each solution names its output file (a path relative to the JSON file, or a URL), and may report a method name, discretization parameters, self-reported wall-clock time, hardware, a link to the source code, and a link to a visualization. The submission is made by pull request (or by direct push, for collaborators) to the solutions repository. See that repository's README for the details.
19+Solutions are not stored here. A solution repository is any public repository on this vault containing `solutions/<challenge-id>/<submission-id>.json` files, validated against [schema/solution.schema.json](schema/solution.schema.json), with the output files beside them or at URLs. One JSON file holds one or more solutions from one submitter for one challenge. Each solution names its wav, and may report a method name, discretization parameters, self-reported wall-clock time, hardware, a link to the source code, a link to a visualization, and an optional export of the solution field for visualization. What else a solution repository contains, such as solver code or its own visualization site, is up to its authors. [challenges/string-solutions](https://vault1.magland.org/challenges/string-solutions) is the first one and serves as an example.
20+
21+The repositories that take part in the comparison are listed in [registry/solution-repos.json](registry/solution-repos.json), one entry per repository with its branch and the path of its `solutions` directory. To register a repository, open a pull request adding an entry. The comparison workflow clones every registered repository at the head of its branch, so a push to a solution repository is picked up the next time the workflow runs: on any push to this repository, or when it is dispatched by hand (`mochi workflow run .mochi/workflows/results.yml --repo challenges/string`). Nothing in the vault triggers across repositories.
2022
2123 ## How solutions are scored
2224
@@ -26,16 +28,17 @@ The reference is chosen as follows. Within a submission, the submitter may asser
2628
2729 The rule has a clear limitation: it is only as good as the asserted orderings. A submission whose sequence is not in fact monotone (for example, a scheme with two error terms of opposite sign that cancel at one resolution) makes a false assertion, and a candidate that happens to reproduce that false pair can be preferred over a more accurate one. With only two submissions this cannot be detected automatically. The results page therefore shows, for every candidate, which asserted pairs it fails to reproduce, so that a reader can judge the choice, and the maintainers can override it with `reference_solution`. More independent submissions make the rule more reliable.
2830
29-The results page lists every solution with its errors relative to the reference, its reported time and hardware, an audio player, and links to code and visualizations. It also reports which asserted orderings the chosen reference does or does not reproduce, and the full matrix of pairwise errors.
31+The results page lists every solution with its errors relative to the reference, its reported time and hardware, an audio player, and links to the submission file, its repository, the source code, the exported fields, and the submitter's visualization. It also reports which asserted orderings the chosen reference does or does not reproduce, and the full matrix of pairwise errors. Submissions are identified as `<repository>/<submission-id>` so that identifiers from different repositories cannot collide.
3032
3133 ## Tools
3234
33-The `tools/` directory holds a small Python package, `challenge_tools`, that validates submissions, computes the error measures, selects the reference, and writes the static results site. The solutions repository runs it in a workflow on every push.
35+The `tools/` directory holds a Python package, `challenge_tools`, that fetches the registered repositories, validates submissions, computes the error measures, selects the reference, and writes the static comparison site. The workflow in `.mochi/workflows/results.yml` runs it on every push and on dispatch. Solution repositories are expected to run only the validator.
3436
3537 ```
3638 pip install .
37-challenge-tools validate --challenges challenges --solutions ../string-solutions/solutions
38-challenge-tools build --challenges challenges --solutions ../string-solutions/solutions --out site
39+challenge-tools fetch --registry registry/solution-repos.json --clones clones
40+challenge-tools validate --challenges challenges --registry registry/solution-repos.json --clones clones
41+challenge-tools build --challenges challenges --registry registry/solution-repos.json --clones clones --out site
3942 ```
4043
41-The error measures are implemented in `tools/challenge_tools/metrics.py` and the reference rule in `tools/challenge_tools/reference.py`; the challenge README is the specification and the code is intended to follow it exactly.
44+A single local directory can stand in for the registry during development with `--solutions <dir>` in place of `--registry` and `--clones`. The error measures are implemented in `tools/challenge_tools/metrics.py` and the reference rule in `tools/challenge_tools/reference.py`; the challenge README is the specification and the code is intended to follow it exactly.
challenges/string-pluck-v1/README.mdmodified+4−0View file
@@ -90,6 +90,10 @@ The spectrogram error ignores phase within a frame and is less sensitive than th
9090
9191 There is no externally supplied ground truth. The reference is chosen automatically from the submitted solutions by the procedure described in the [collection README](../../README.md), unless `reference_solution` in challenge.json names one explicitly. For this particular challenge the modal series in the appendix gives an essentially exact answer, which is expected to be selected.
9292
93+## Optional field export
94+
95+A submission may additionally provide the displacement field for visualization. This is not scored. The file is HDF5 with three datasets: `x`, float64, shape (201,), the positions $x_i = i L / 200$ in meters; `t`, float64, shape (4001,), the times $t_j = j / 2000$ in seconds; and `u`, float32, shape (4001, 201), with `u[j, i]` $= u(x_i, t_j)$ in meters. Root attributes `challenge_id`, `solution_id`, `quantity` (`"transverse displacement"`) and `units` (`"m"`) are recommended. The grid is stated under `fields` in challenge.json. A file of this size is about 3 MB uncompressed and can live in the solution repository; gzip compression inside the file is welcome.
96+
9397 ## Appendix: modal series solution
9498
9599 This appendix is provided for convenience and verification; it is not part of the challenge definition, and a submission may use any method.
challenges/string-pluck-v1/challenge.jsonmodified+10−1View file
@@ -31,5 +31,14 @@
3131 "hop": 512
3232 }
3333 },
34- "reference_solution": null
34+ "reference_solution": null,
35+ "fields": {
36+ "format": "hdf5",
37+ "quantity": "transverse displacement u(x, t)",
38+ "units": "m",
39+ "nx": 201,
40+ "nt": 4001,
41+ "frame_rate_hz": 2000,
42+ "dtype": "float32"
43+ }
3544 }
pyproject.tomlmodified+1−1View file
@@ -7,7 +7,7 @@ name = "challenge-tools"
77 version = "0.1.0"
88 description = "Validation, scoring, and results-site generation for the instrument simulation challenges"
99 requires-python = ">=3.10"
10-dependencies = ["numpy", "scipy", "jsonschema"]
10+dependencies = ["numpy", "scipy", "jsonschema", "h5py"]
1111
1212 [project.scripts]
1313 challenge-tools = "challenge_tools.cli:main"
registry/solution-repos.jsonadded+10−0View file
@@ -0,0 +1,10 @@
1+{
2+ "repos": [
3+ {
4+ "repo": "challenges/string-solutions",
5+ "branch": "main",
6+ "path": "solutions",
7+ "description": "Example submissions by the challenge authors: modal series and second-order finite differences."
8+ }
9+ ]
10+}
schema/solution.schema.jsonmodified+94−19View file
@@ -4,10 +4,18 @@
44 "title": "Challenge solution submission",
55 "description": "One submission file: a set of solutions to one challenge from one submitter, with an optional accuracy partial order among them.",
66 "type": "object",
7- "required": ["schema_version", "submission_id", "challenge_id", "submitter", "solutions"],
7+ "required": [
8+ "schema_version",
9+ "submission_id",
10+ "challenge_id",
11+ "submitter",
12+ "solutions"
13+ ],
814 "additionalProperties": false,
915 "properties": {
10- "schema_version": { "const": 1 },
16+ "schema_version": {
17+ "const": 1
18+ },
1119 "submission_id": {
1220 "type": "string",
1321 "pattern": "^[a-z0-9][a-z0-9-]*$",
@@ -20,24 +28,49 @@
2028 },
2129 "submitter": {
2230 "type": "object",
23- "required": ["name"],
31+ "required": [
32+ "name"
33+ ],
2434 "additionalProperties": false,
2535 "properties": {
26- "name": { "type": "string" },
27- "email": { "type": "string" },
28- "url": { "type": "string", "format": "uri" }
36+ "name": {
37+ "type": "string"
38+ },
39+ "email": {
40+ "type": "string"
41+ },
42+ "url": {
43+ "type": "string",
44+ "format": "uri"
45+ }
2946 }
3047 },
31- "date": { "type": "string", "format": "date" },
32- "description": { "type": "string" },
33- "code_url": { "type": "string", "format": "uri", "description": "Source code for the method, applying to every solution unless overridden." },
34- "viz_url": { "type": "string", "format": "uri", "description": "A web application visualizing the solutions." },
48+ "date": {
49+ "type": "string",
50+ "format": "date"
51+ },
52+ "description": {
53+ "type": "string"
54+ },
55+ "code_url": {
56+ "type": "string",
57+ "format": "uri",
58+ "description": "Source code for the method, applying to every solution unless overridden."
59+ },
60+ "viz_url": {
61+ "type": "string",
62+ "format": "uri",
63+ "description": "A web application visualizing the solutions."
64+ },
3565 "solutions": {
3666 "type": "array",
3767 "minItems": 1,
3868 "items": {
3969 "type": "object",
40- "required": ["solution_id", "output"],
70+ "required": [
71+ "solution_id",
72+ "output"
73+ ],
4174 "additionalProperties": false,
4275 "properties": {
4376 "solution_id": {
@@ -49,18 +82,60 @@
4982 "type": "string",
5083 "description": "Path to the wav file relative to the directory containing this JSON file, or an http(s) URL."
5184 },
52- "method": { "type": "string", "description": "Short name of the numerical method." },
53- "parameters": { "type": "object", "description": "Discretization parameters, free form." },
54- "timing_seconds": { "type": "number", "minimum": 0, "description": "Self-reported wall-clock time to produce the output, excluding file I/O." },
55- "hardware": { "type": "string" },
56- "code_url": { "type": "string", "format": "uri" },
57- "viz_url": { "type": "string", "format": "uri" },
85+ "method": {
86+ "type": "string",
87+ "description": "Short name of the numerical method."
88+ },
89+ "parameters": {
90+ "type": "object",
91+ "description": "Discretization parameters, free form."
92+ },
93+ "timing_seconds": {
94+ "type": "number",
95+ "minimum": 0,
96+ "description": "Self-reported wall-clock time to produce the output, excluding file I/O."
97+ },
98+ "hardware": {
99+ "type": "string"
100+ },
101+ "code_url": {
102+ "type": "string",
103+ "format": "uri"
104+ },
105+ "viz_url": {
106+ "type": "string",
107+ "format": "uri"
108+ },
58109 "more_accurate_than": {
59110 "type": "array",
60- "items": { "type": "string" },
111+ "items": {
112+ "type": "string"
113+ },
61114 "description": "solution_ids in this same file that the submitter asserts are less accurate than this one."
62115 },
63- "notes": { "type": "string" }
116+ "notes": {
117+ "type": "string"
118+ },
119+ "fields": {
120+ "type": "object",
121+ "required": [
122+ "path"
123+ ],
124+ "additionalProperties": false,
125+ "description": "Optional export of the solution field on the grid recommended by the challenge, for visualization. Not scored.",
126+ "properties": {
127+ "path": {
128+ "type": "string",
129+ "description": "Path relative to the directory containing this JSON file, or an http(s) URL."
130+ },
131+ "format": {
132+ "const": "hdf5"
133+ },
134+ "description": {
135+ "type": "string"
136+ }
137+ }
138+ }
64139 }
65140 }
66141 }
tools/challenge_tools/build.pymodified+40−23View file
@@ -1,4 +1,4 @@
1-"""Compute all scores and write the results site."""
1+"""Compute all scores and write the comparison site."""
22 from __future__ import annotations
33
44 import datetime as dt
@@ -9,7 +9,7 @@ from pathlib import Path
99
1010 import numpy as np
1111
12-from .loading import load_challenges, load_submissions
12+from .loading import Source, load_challenges, load_submissions, public_fields
1313 from .metrics import compute_metric, metric_names
1414 from .reference import collect_order, select_reference
1515 from .wav import read_wav, resolve_output
@@ -24,19 +24,27 @@ def score_challenge(challenge: dict, submissions: list[dict], out_audio: Path |
2424 signals: dict[str, np.ndarray] = {}
2525 rows: dict[str, dict] = {}
2626 for sub in submissions:
27+ src: Source = sub["_source"]
2728 for sol in sub["solutions"]:
28- key = f"{sub['submission_id']}/{sol['solution_id']}"
29+ key = f"{sub['_key']}/{sol['solution_id']}"
2930 p = resolve_output(sol["output"], sub["_path"].parent)
3031 signals[key] = read_wav(p, spec)
3132 audio_rel = None
3233 if out_audio is not None:
33- dest = out_audio / sub["submission_id"] / f"{sol['solution_id']}.wav"
34+ dest = out_audio / sub["_key"] / f"{sol['solution_id']}.wav"
3435 dest.parent.mkdir(parents=True, exist_ok=True)
3536 shutil.copyfile(p, dest)
36- audio_rel = f"audio/{sub['submission_id']}/{sol['solution_id']}.wav"
37+ audio_rel = f"audio/{sub['_key']}/{sol['solution_id']}.wav"
38+ fields_url = None
39+ if "fields" in sol:
40+ fpath = sol["fields"]["path"]
41+ fields_url = fpath if fpath.startswith("http") else src.blob_url(sub["_path"].parent / fpath)
3742 rows[key] = {
3843 "key": key,
44+ "source": src.label,
45+ "source_url": src.url,
3946 "submission_id": sub["submission_id"],
47+ "submission_url": src.blob_url(sub["_path"]),
4048 "solution_id": sol["solution_id"],
4149 "submitter": sub["submitter"]["name"],
4250 "date": sub.get("date"),
@@ -46,7 +54,8 @@ def score_challenge(challenge: dict, submissions: list[dict], out_audio: Path |
4654 "hardware": sol.get("hardware"),
4755 "code_url": sol.get("code_url") or sub.get("code_url"),
4856 "viz_url": sol.get("viz_url") or sub.get("viz_url"),
49- "more_accurate_than": [f"{sub['submission_id']}/{w}" for w in sol.get("more_accurate_than", [])],
57+ "fields_url": fields_url,
58+ "more_accurate_than": [f"{sub['_key']}/{w}" for w in sol.get("more_accurate_than", [])],
5059 "notes": sol.get("notes"),
5160 "audio": audio_rel,
5261 "peak": float(np.max(np.abs(signals[key]))),
@@ -70,7 +79,6 @@ def score_challenge(challenge: dict, submissions: list[dict], out_audio: Path |
7079 row["errors"] = {name: pairwise[name][key][ref] for name in names} if ref else {}
7180 row["is_reference"] = key == ref
7281
73- # Check every asserted pair against the chosen reference, for the report.
7482 violations = []
7583 if ref:
7684 for worse, better, _sid in info.pairs:
@@ -91,7 +99,8 @@ def score_challenge(challenge: dict, submissions: list[dict], out_audio: Path |
9199 "solutions": ordered,
92100 "pairwise": pairwise,
93101 "submissions": [
94- {k: v for k, v in sub.items() if not k.startswith("_") and k != "solutions"} for sub in submissions
102+ {**{k: v for k, v in public_fields(sub).items() if k != "solutions"}, "key": sub["_key"], "source": sub["_source"].label, "source_url": sub["_source"].url}
103+ for sub in submissions
95104 ],
96105 }
97106
@@ -102,7 +111,7 @@ _CSS = """
102111 :root { --fg: #1b1b1b; --bg: #ffffff; --muted: #666; --line: #ddd; --accent: #2a6db0; --ref: #e8f4e8; }
103112 @media (prefers-color-scheme: dark) { :root { --fg: #e6e6e6; --bg: #151515; --muted: #999; --line: #333; --accent: #7ab3e8; --ref: #1f3320; } }
104113 body { font-family: system-ui, sans-serif; color: var(--fg); background: var(--bg); margin: 0; padding: 16px; line-height: 1.45; }
105-main { max-width: 1200px; margin: 0 auto; }
114+main { max-width: 1300px; margin: 0 auto; }
106115 a { color: var(--accent); }
107116 table { border-collapse: collapse; width: 100%; font-size: 0.92em; }
108117 th, td { border-bottom: 1px solid var(--line); padding: 6px 8px; text-align: left; vertical-align: top; }
@@ -136,19 +145,21 @@ def _link(url, text):
136145 def render_challenge_page(res: dict, challenge: dict, links: dict) -> str:
137146 names = res["metrics"]
138147 primary = res["primary_metric"]
139- head = "".join(
140- f"<th>{html.escape(n)}{' (primary)' if n == primary else ''}</th>" for n in names
141- )
148+ head = "".join(f"<th>{html.escape(n)}{' (primary)' if n == primary else ''}</th>" for n in names)
142149 body = []
143150 for r in res["solutions"]:
144151 errs = "".join(f'<td class="num">{_fmt(r["errors"].get(n))}</td>' for n in names)
145152 params = html.escape(json.dumps(r["parameters"])) if r["parameters"] else ""
146- mat = ", ".join(html.escape(k.split("/", 1)[1]) for k in r["more_accurate_than"])
153+ mat = ", ".join(html.escape(k.rsplit("/", 1)[1]) for k in r["more_accurate_than"])
147154 audio = f'<audio controls preload="none" src="{html.escape(r["audio"])}"></audio>' if r["audio"] else ""
148- linkbits = " ".join(x for x in [_link(r["code_url"], "code"), _link(r["viz_url"], "viz"), _link(r["audio"], "wav")] if x)
155+ linkbits = " ".join(x for x in [
156+ _link(r["audio"], "wav"), _link(r["fields_url"], "fields"), _link(r["code_url"], "code"), _link(r["viz_url"], "viz"),
157+ ] if x)
158+ sub_cell = _link(r["submission_url"], r["submission_id"]) if r["submission_url"] else html.escape(r["submission_id"])
159+ src_cell = _link(r["source_url"], r["source"]) if r["source"] else ""
149160 body.append(
150161 f'<tr class="{"ref" if r["is_reference"] else ""}">'
151- f'<td>{html.escape(r["submission_id"])}<br><span class="muted">{html.escape(r["submitter"])}</span></td>'
162+ f'<td>{sub_cell}<br><span class="muted">{src_cell}{" · " if src_cell else ""}{html.escape(r["submitter"])}</span></td>'
152163 f'<td>{html.escape(r["solution_id"])}{" <b>(reference)</b>" if r["is_reference"] else ""}</td>'
153164 f'<td>{html.escape(r["method"] or "")}<br><span class="muted"><code>{params}</code></span></td>'
154165 f"{errs}"
@@ -191,7 +202,7 @@ def render_challenge_page(res: dict, challenge: dict, links: dict) -> str:
191202 <p><a href="../">All challenges</a></p>
192203 <h1>{html.escape(res["challenge_id"])}</h1>
193204 <p>{html.escape(res.get("title") or "")}. {"<a href='" + html.escape(spec_link) + "'>Challenge specification</a>. " if spec_link else ""}<a href="results.json">results.json</a>. Generated {html.escape(res["generated"])}.</p>
194-<p>Errors are relative to the reference solution <b>{html.escape(res["reference"] or "none")}</b>, chosen by rule <i>{html.escape(sel["rule"])}</i>. Rows are sorted by the primary metric.</p>
205+<p>Errors are relative to the reference solution <b>{html.escape(res["reference"] or "none")}</b>, chosen by rule <i>{html.escape(sel["rule"])}</i>. Rows are sorted by the primary metric. Submissions are named as <i>repository/submission</i>.</p>
195206 <div class="wrap"><table>
196207 <tr><th>Submission</th><th>Solution</th><th>Method</th>{head}<th>Time (s)</th><th>Hardware</th><th>Claimed more accurate than</th><th>Output</th></tr>
197208 {"".join(body)}
@@ -206,28 +217,34 @@ def render_challenge_page(res: dict, challenge: dict, links: dict) -> str:
206217 """
207218
208219
209-def render_index(results: list[dict], links: dict) -> str:
220+def render_index(results: list[dict], sources: list[Source], links: dict) -> str:
210221 items = "".join(
211222 f'<li><a href="{html.escape(r["challenge_id"])}/">{html.escape(r["challenge_id"])}</a>: {html.escape(r.get("title") or "")} '
212223 f'<span class="muted">({len(r["solutions"])} solutions, reference {html.escape(r["reference"] or "none")})</span></li>'
213224 for r in results
214225 )
226+ srcs = "".join(
227+ f"<li>{_link(s.url, s.repo or s.label or str(s.dir)) if s.url else html.escape(str(s.dir))}"
228+ f"{' <span class=muted>(' + html.escape(s.branch) + ')</span>' if s.branch else ''}"
229+ f"{': ' + html.escape(s.description) if s.description else ''}</li>"
230+ for s in sources
231+ )
215232 repo = links.get("challenge_repo")
216- sols = links.get("solutions_repo")
217233 return f"""<!doctype html>
218234 <html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1">
219235 <title>Challenge results</title><style>{_CSS}</style></head>
220236 <body><main>
221237 <h1>Challenge results</h1>
222-<p>{"Specifications: <a href='" + html.escape(repo) + "'>" + html.escape(repo) + "</a>. " if repo else ""}{"Solutions: <a href='" + html.escape(sols) + "'>" + html.escape(sols) + "</a>." if sols else ""}</p>
223-<ul>{items}</ul>
238+<p>{"Specifications and registry: <a href='" + html.escape(repo) + "'>" + html.escape(repo) + "</a>." if repo else ""}</p>
239+<h2>Challenges</h2><ul>{items}</ul>
240+<h2>Registered solution repositories</h2><ul>{srcs}</ul>
224241 </main></body></html>
225242 """
226243
227244
228-def build(challenges_dir: Path, solutions_dir: Path, out: Path, challenge_repo_url: str | None = None, solutions_repo_url: str | None = None) -> list[dict]:
245+def build(challenges_dir: Path, sources: list[Source], out: Path, challenge_repo_url: str | None = None) -> list[dict]:
229246 challenges = load_challenges(challenges_dir)
230- submissions = load_submissions(solutions_dir)
247+ submissions = load_submissions(sources)
231248 out.mkdir(parents=True, exist_ok=True)
232249 results = []
233250 for cid, ch in challenges.items():
@@ -244,5 +261,5 @@ def build(challenges_dir: Path, solutions_dir: Path, out: Path, challenge_repo_u
244261 links = {"spec": f"{challenge_repo_url}/blob/main/challenges/{cid}/README.md" if challenge_repo_url else None}
245262 (cdir / "index.html").write_text(render_challenge_page(res, ch, links))
246263 results.append(res)
247- (out / "index.html").write_text(render_index(results, {"challenge_repo": challenge_repo_url, "solutions_repo": solutions_repo_url}))
264+ (out / "index.html").write_text(render_index(results, sources, {"challenge_repo": challenge_repo_url}))
248265 return results
tools/challenge_tools/cli.pymodified+41−11View file
@@ -4,47 +4,77 @@ import argparse
44 import sys
55 from pathlib import Path
66
7+from .loading import Source, fetch_registry, local_source, registry_sources
8+
9+DEFAULT_VAULT = "https://vault1.magland.org"
10+
11+
12+def _add_source_args(p: argparse.ArgumentParser) -> None:
13+ g = p.add_argument_group("solution sources (one of)")
14+ g.add_argument("--solutions", type=Path, help="a local directory containing <challenge-id>/<submission-id>.json")
15+ g.add_argument("--registry", type=Path, help="registry JSON listing solution repositories")
16+ g.add_argument("--clones", type=Path, default=Path("clones"), help="where 'fetch' put the registered repositories")
17+ g.add_argument("--vault-url", default=DEFAULT_VAULT)
18+
19+
20+def _sources(args) -> list[Source]:
21+ if args.solutions is not None and args.registry is not None:
22+ raise SystemExit("give either --solutions or --registry, not both")
23+ if args.solutions is not None:
24+ return [local_source(args.solutions)]
25+ if args.registry is not None:
26+ return registry_sources(args.registry, args.clones, args.vault_url)
27+ raise SystemExit("one of --solutions or --registry is required")
28+
729
830 def main(argv=None):
9- ap = argparse.ArgumentParser(prog="challenge-tools", description="Validate submissions and build the results site.")
31+ ap = argparse.ArgumentParser(prog="challenge-tools", description="Validate submissions and build the comparison site.")
1032 sp = ap.add_subparsers(dest="cmd", required=True)
1133
34+ f = sp.add_parser("fetch", help="clone or update the registered solution repositories")
35+ f.add_argument("--registry", required=True, type=Path)
36+ f.add_argument("--clones", type=Path, default=Path("clones"))
37+ f.add_argument("--vault-url", default=DEFAULT_VAULT)
38+
1239 v = sp.add_parser("validate", help="check submission files and their outputs")
1340 v.add_argument("--challenges", required=True, type=Path, help="directory containing <challenge-id>/challenge.json")
14- v.add_argument("--solutions", required=True, type=Path, help="directory containing <challenge-id>/<submission-id>.json")
15- v.add_argument("--no-outputs", action="store_true", help="skip reading the wav files")
41+ _add_source_args(v)
42+ v.add_argument("--no-outputs", action="store_true", help="skip reading the wav and fields files")
1643
17- b = sp.add_parser("build", help="compute scores and write the static results site")
44+ b = sp.add_parser("build", help="compute scores and write the static comparison site")
1845 b.add_argument("--challenges", required=True, type=Path)
19- b.add_argument("--solutions", required=True, type=Path)
46+ _add_source_args(b)
2047 b.add_argument("--out", required=True, type=Path)
2148 b.add_argument("--challenge-repo-url", default=None)
22- b.add_argument("--solutions-repo-url", default=None)
2349
2450 args = ap.parse_args(argv)
51+ if args.cmd == "fetch":
52+ fetch_registry(args.registry, args.clones, args.vault_url)
53+ return 0
2554 if args.cmd == "validate":
2655 from .validate import validate
2756
28- problems = validate(args.challenges, args.solutions, check_outputs=not args.no_outputs)
57+ problems = validate(args.challenges, _sources(args), check_outputs=not args.no_outputs)
2958 for p in problems:
3059 print("PROBLEM:", p, file=sys.stderr)
3160 print(f"{len(problems)} problem(s)")
3261 return 1 if problems else 0
3362 if args.cmd == "build":
34- from .validate import validate
3563 from .build import build
64+ from .validate import validate
3665
37- problems = validate(args.challenges, args.solutions)
66+ sources = _sources(args)
67+ problems = validate(args.challenges, sources)
3868 if problems:
3969 for p in problems:
4070 print("PROBLEM:", p, file=sys.stderr)
4171 return 1
42- results = build(args.challenges, args.solutions, args.out, args.challenge_repo_url, args.solutions_repo_url)
72+ results = build(args.challenges, sources, args.out, args.challenge_repo_url)
4373 for r in results:
4474 print(f"{r['challenge_id']}: {len(r['solutions'])} solutions, reference = {r['reference']}")
4575 for s in r["solutions"]:
4676 errs = " ".join(f"{k}={v:.3e}" for k, v in s["errors"].items())
47- print(f" {s['key']:40s} {errs} t={s['timing_seconds']}")
77+ print(f" {s['key']:50s} {errs} t={s['timing_seconds']}")
4878 return 0
4979 return 2
5080
tools/challenge_tools/loading.pymodified+105−9View file
@@ -1,10 +1,34 @@
1-"""Loading challenge definitions and submission files."""
1+"""Loading challenge definitions, solution sources, and submission files."""
22 from __future__ import annotations
33
44 import json
5+import subprocess
6+from dataclasses import dataclass
57 from pathlib import Path
68
79
10+@dataclass
11+class Source:
12+ """One place solutions come from: a registered repository clone, or a local directory."""
13+
14+ label: str | None # short name used to namespace submission ids; None for a single local directory
15+ dir: Path # directory holding <challenge-id>/<submission-id>.json
16+ repo: str | None = None # "<collection>/<repo>"
17+ url: str | None = None # web URL of the repository
18+ branch: str | None = None
19+ root: Path | None = None # clone root, to express files as repository-relative paths
20+ description: str | None = None
21+
22+ def blob_url(self, path: Path) -> str | None:
23+ if self.url is None or self.root is None:
24+ return None
25+ try:
26+ rel = Path(path).resolve().relative_to(self.root.resolve())
27+ except ValueError:
28+ return None
29+ return f"{self.url}/blob/{self.branch or 'main'}/{rel.as_posix()}"
30+
31+
832 def load_challenges(challenges_dir: Path) -> dict[str, dict]:
933 out = {}
1034 for p in sorted(Path(challenges_dir).glob("*/challenge.json")):
@@ -15,13 +39,85 @@ def load_challenges(challenges_dir: Path) -> dict[str, dict]:
1539 return out
1640
1741
18-def load_submissions(solutions_dir: Path) -> list[dict]:
19- """Return every submission JSON under solutions/<challenge-id>/*.json, with _path and _dir_challenge set."""
42+def read_registry(registry_path: Path) -> list[dict]:
43+ with open(registry_path) as f:
44+ reg = json.load(f)
45+ repos = reg["repos"]
46+ for r in repos:
47+ r.setdefault("branch", "main")
48+ r.setdefault("path", "solutions")
49+ r["label"] = r.get("label") or r["repo"].split("/")[-1]
50+ labels = [r["label"] for r in repos]
51+ if len(set(labels)) != len(labels):
52+ raise ValueError("registry labels are not unique; give colliding entries an explicit 'label'")
53+ return repos
54+
55+
56+def clone_dir(clones: Path, entry: dict) -> Path:
57+ return Path(clones) / entry["label"]
58+
59+
60+def fetch_registry(registry_path: Path, clones: Path, vault_url: str) -> list[Source]:
61+ """Clone or update every registered repository (shallow, one branch). Anonymous, so public repositories only."""
62+ sources = []
63+ clones.mkdir(parents=True, exist_ok=True)
64+ for entry in read_registry(registry_path):
65+ dest = clone_dir(clones, entry)
66+ remote = f"{vault_url.rstrip('/')}/{entry['repo']}"
67+ if (dest / ".git").exists():
68+ subprocess.run(["git", "-C", str(dest), "fetch", "--depth", "1", "origin", entry["branch"]], check=True)
69+ subprocess.run(["git", "-C", str(dest), "checkout", "-q", "--force", "FETCH_HEAD"], check=True)
70+ else:
71+ subprocess.run(["git", "clone", "-q", "--depth", "1", "--branch", entry["branch"], remote, str(dest)], check=True)
72+ sha = subprocess.run(["git", "-C", str(dest), "rev-parse", "HEAD"], check=True, capture_output=True, text=True).stdout.strip()
73+ print(f"{entry['repo']}@{entry['branch']} -> {dest} ({sha[:10]})")
74+ sources.append(source_from_entry(entry, clones, vault_url))
75+ return sources
76+
77+
78+def source_from_entry(entry: dict, clones: Path, vault_url: str) -> Source:
79+ root = clone_dir(clones, entry)
80+ return Source(
81+ label=entry["label"],
82+ dir=root / entry["path"],
83+ repo=entry["repo"],
84+ url=f"{vault_url.rstrip('/')}/{entry['repo']}",
85+ branch=entry["branch"],
86+ root=root,
87+ description=entry.get("description"),
88+ )
89+
90+
91+def registry_sources(registry_path: Path, clones: Path, vault_url: str) -> list[Source]:
92+ return [source_from_entry(e, clones, vault_url) for e in read_registry(registry_path)]
93+
94+
95+def local_source(solutions_dir: Path) -> Source:
96+ return Source(label=None, dir=Path(solutions_dir))
97+
98+
99+def submission_key(sub: dict) -> str:
100+ label = sub.get("_source").label if sub.get("_source") else None
101+ return f"{label}/{sub['submission_id']}" if label else sub["submission_id"]
102+
103+
104+def load_submissions(sources: list[Source]) -> list[dict]:
105+ """Every submission JSON under <source.dir>/<challenge-id>/*.json, with _source, _path, _dir_challenge, _key set."""
20106 out = []
21- for p in sorted(Path(solutions_dir).glob("*/*.json")):
22- with open(p) as f:
23- sub = json.load(f)
24- sub["_path"] = p
25- sub["_dir_challenge"] = p.parent.name
26- out.append(sub)
107+ for src in sources:
108+ if not src.dir.exists():
109+ print(f"warning: solutions directory {src.dir} does not exist")
110+ continue
111+ for p in sorted(src.dir.glob("*/*.json")):
112+ with open(p) as f:
113+ sub = json.load(f)
114+ sub["_source"] = src
115+ sub["_path"] = p
116+ sub["_dir_challenge"] = p.parent.name
117+ sub["_key"] = submission_key(sub)
118+ out.append(sub)
27119 return out
120+
121+
122+def public_fields(sub: dict) -> dict:
123+ return {k: v for k, v in sub.items() if not k.startswith("_")}
tools/challenge_tools/reference.pymodified+2−2View file
@@ -15,7 +15,7 @@ from dataclasses import dataclass, field
1515
1616 @dataclass
1717 class OrderInfo:
18- # (worse, better, submission_id) triples over global solution keys "submission/solution"
18+ # (worse, better, submission_key) triples over global solution keys "[source/]submission/solution"
1919 pairs: list[tuple[str, str, str]] = field(default_factory=list)
2020 # global key -> submission id
2121 submission_of: dict[str, str] = field(default_factory=dict)
@@ -24,7 +24,7 @@ class OrderInfo:
2424 def collect_order(submissions: list[dict]) -> OrderInfo:
2525 info = OrderInfo()
2626 for sub in submissions:
27- sid = sub["submission_id"]
27+ sid = sub.get("_key", sub["submission_id"])
2828 for sol in sub["solutions"]:
2929 key = f"{sid}/{sol['solution_id']}"
3030 info.submission_of[key] = sid
tools/challenge_tools/solution.schema.jsonmodified+94−19View file
@@ -4,10 +4,18 @@
44 "title": "Challenge solution submission",
55 "description": "One submission file: a set of solutions to one challenge from one submitter, with an optional accuracy partial order among them.",
66 "type": "object",
7- "required": ["schema_version", "submission_id", "challenge_id", "submitter", "solutions"],
7+ "required": [
8+ "schema_version",
9+ "submission_id",
10+ "challenge_id",
11+ "submitter",
12+ "solutions"
13+ ],
814 "additionalProperties": false,
915 "properties": {
10- "schema_version": { "const": 1 },
16+ "schema_version": {
17+ "const": 1
18+ },
1119 "submission_id": {
1220 "type": "string",
1321 "pattern": "^[a-z0-9][a-z0-9-]*$",
@@ -20,24 +28,49 @@
2028 },
2129 "submitter": {
2230 "type": "object",
23- "required": ["name"],
31+ "required": [
32+ "name"
33+ ],
2434 "additionalProperties": false,
2535 "properties": {
26- "name": { "type": "string" },
27- "email": { "type": "string" },
28- "url": { "type": "string", "format": "uri" }
36+ "name": {
37+ "type": "string"
38+ },
39+ "email": {
40+ "type": "string"
41+ },
42+ "url": {
43+ "type": "string",
44+ "format": "uri"
45+ }
2946 }
3047 },
31- "date": { "type": "string", "format": "date" },
32- "description": { "type": "string" },
33- "code_url": { "type": "string", "format": "uri", "description": "Source code for the method, applying to every solution unless overridden." },
34- "viz_url": { "type": "string", "format": "uri", "description": "A web application visualizing the solutions." },
48+ "date": {
49+ "type": "string",
50+ "format": "date"
51+ },
52+ "description": {
53+ "type": "string"
54+ },
55+ "code_url": {
56+ "type": "string",
57+ "format": "uri",
58+ "description": "Source code for the method, applying to every solution unless overridden."
59+ },
60+ "viz_url": {
61+ "type": "string",
62+ "format": "uri",
63+ "description": "A web application visualizing the solutions."
64+ },
3565 "solutions": {
3666 "type": "array",
3767 "minItems": 1,
3868 "items": {
3969 "type": "object",
40- "required": ["solution_id", "output"],
70+ "required": [
71+ "solution_id",
72+ "output"
73+ ],
4174 "additionalProperties": false,
4275 "properties": {
4376 "solution_id": {
@@ -49,18 +82,60 @@
4982 "type": "string",
5083 "description": "Path to the wav file relative to the directory containing this JSON file, or an http(s) URL."
5184 },
52- "method": { "type": "string", "description": "Short name of the numerical method." },
53- "parameters": { "type": "object", "description": "Discretization parameters, free form." },
54- "timing_seconds": { "type": "number", "minimum": 0, "description": "Self-reported wall-clock time to produce the output, excluding file I/O." },
55- "hardware": { "type": "string" },
56- "code_url": { "type": "string", "format": "uri" },
57- "viz_url": { "type": "string", "format": "uri" },
85+ "method": {
86+ "type": "string",
87+ "description": "Short name of the numerical method."
88+ },
89+ "parameters": {
90+ "type": "object",
91+ "description": "Discretization parameters, free form."
92+ },
93+ "timing_seconds": {
94+ "type": "number",
95+ "minimum": 0,
96+ "description": "Self-reported wall-clock time to produce the output, excluding file I/O."
97+ },
98+ "hardware": {
99+ "type": "string"
100+ },
101+ "code_url": {
102+ "type": "string",
103+ "format": "uri"
104+ },
105+ "viz_url": {
106+ "type": "string",
107+ "format": "uri"
108+ },
58109 "more_accurate_than": {
59110 "type": "array",
60- "items": { "type": "string" },
111+ "items": {
112+ "type": "string"
113+ },
61114 "description": "solution_ids in this same file that the submitter asserts are less accurate than this one."
62115 },
63- "notes": { "type": "string" }
116+ "notes": {
117+ "type": "string"
118+ },
119+ "fields": {
120+ "type": "object",
121+ "required": [
122+ "path"
123+ ],
124+ "additionalProperties": false,
125+ "description": "Optional export of the solution field on the grid recommended by the challenge, for visualization. Not scored.",
126+ "properties": {
127+ "path": {
128+ "type": "string",
129+ "description": "Path relative to the directory containing this JSON file, or an http(s) URL."
130+ },
131+ "format": {
132+ "const": "hdf5"
133+ },
134+ "description": {
135+ "type": "string"
136+ }
137+ }
138+ }
64139 }
65140 }
66141 }
tools/challenge_tools/validate.pymodified+38−7View file
@@ -7,7 +7,7 @@ from pathlib import Path
77
88 import jsonschema
99
10-from .loading import load_challenges, load_submissions
10+from .loading import Source, load_challenges, load_submissions, public_fields
1111 from .wav import read_wav, resolve_output
1212
1313
@@ -33,7 +33,28 @@ def _acyclic(edges: list[tuple[str, str]], nodes: list[str]) -> bool:
3333 return all(state[n] == 2 or visit(n) for n in nodes)
3434
3535
36-def validate(challenges_dir: Path, solutions_dir: Path, check_outputs: bool = True) -> list[str]:
36+def check_fields_file(path: Path, challenge: dict) -> list[str]:
37+ """Open an optional HDF5 field export and check the datasets against the challenge's grid."""
38+ import h5py
39+
40+ problems = []
41+ spec = challenge.get("fields")
42+ with h5py.File(path, "r") as f:
43+ for name in ("x", "t", "u"):
44+ if name not in f:
45+ problems.append(f"fields file lacks dataset {name!r}")
46+ if problems:
47+ return problems
48+ nx, nt = f["x"].shape[0], f["t"].shape[0]
49+ if f["u"].shape != (nt, nx):
50+ problems.append(f"fields dataset u has shape {f['u'].shape}, expected {(nt, nx)}")
51+ if spec:
52+ if nx != spec["nx"] or nt != spec["nt"]:
53+ problems.append(f"fields grid is {nx} x {nt}, challenge recommends nx={spec['nx']} nt={spec['nt']}")
54+ return problems
55+
56+
57+def validate(challenges_dir: Path, sources: list[Source], check_outputs: bool = True) -> list[str]:
3758 """Return a list of problems (empty means valid)."""
3859 problems: list[str] = []
3960 challenges = load_challenges(challenges_dir)
@@ -42,10 +63,10 @@ def validate(challenges_dir: Path, solutions_dir: Path, check_outputs: bool = Tr
4263 schema = _schema()
4364 validator = jsonschema.Draft202012Validator(schema, format_checker=jsonschema.FormatChecker())
4465 seen = set()
45- for sub in load_submissions(solutions_dir):
66+ for sub in load_submissions(sources):
4667 path: Path = sub["_path"]
4768 tag = str(path)
48- errs = sorted(validator.iter_errors({k: v for k, v in sub.items() if not k.startswith("_")}), key=str)
69+ errs = sorted(validator.iter_errors(public_fields(sub)), key=str)
4970 for e in errs:
5071 loc = "/".join(str(x) for x in e.absolute_path)
5172 problems.append(f"{tag}: schema: {loc}: {e.message}")
@@ -58,7 +79,7 @@ def validate(challenges_dir: Path, solutions_dir: Path, check_outputs: bool = Tr
5879 if sub["challenge_id"] not in challenges:
5980 problems.append(f"{tag}: unknown challenge_id {sub['challenge_id']!r}")
6081 continue
61- key = (sub["challenge_id"], sub["submission_id"])
82+ key = (sub["challenge_id"], sub["_key"])
6283 if key in seen:
6384 problems.append(f"{tag}: duplicate submission_id for this challenge")
6485 seen.add(key)
@@ -77,14 +98,24 @@ def validate(challenges_dir: Path, solutions_dir: Path, check_outputs: bool = Tr
7798 if not _acyclic(edges, ids):
7899 problems.append(f"{tag}: more_accurate_than contains a cycle")
79100 if check_outputs:
80- spec = challenges[sub["challenge_id"]]["output"]
101+ challenge = challenges[sub["challenge_id"]]
81102 for s in sub["solutions"]:
82103 try:
83104 p = resolve_output(s["output"], path.parent)
84105 if not p.exists():
85106 problems.append(f"{tag}: {s['solution_id']}: output not found: {s['output']}")
86107 continue
87- read_wav(p, spec)
108+ read_wav(p, challenge["output"])
88109 except Exception as e: # noqa: BLE001
89110 problems.append(f"{tag}: {s['solution_id']}: {e}")
111+ if "fields" in s:
112+ try:
113+ fp = resolve_output(s["fields"]["path"], path.parent)
114+ if not fp.exists():
115+ problems.append(f"{tag}: {s['solution_id']}: fields file not found: {s['fields']['path']}")
116+ else:
117+ for pr in check_fields_file(fp, challenge):
118+ problems.append(f"{tag}: {s['solution_id']}: {pr}")
119+ except Exception as e: # noqa: BLE001
120+ problems.append(f"{tag}: {s['solution_id']}: fields: {e}")
90121 return problems