ddfc839Add string-pluck-v1 challenge, solution schema, and scoring toolsJeremy Magland 1# Instrument simulation challenges: string (tier 0)
03370d1Register solution repositories and build the comparison site here; optional HDF5 field export in the schema and specJeremy Magland 3This repository defines simulation challenges for a damped string, and publishes the comparison of every registered solution. It is the first, and simplest, repository in a collection of challenges for physics-based simulation of musical instruments, in which each challenge is a fully specified continuum model together with a precisely defined output signal and error measures. Solutions live in separate repositories that are registered here; the comparison is at [challenges/string/site](https://vault1.magland.org/challenges/string/site/).
5## Challenges
7| ID | Title | Deliverable |
8| --- | --- | --- |
9| [string-pluck-v1](challenges/string-pluck-v1/README.md) | Plucked damped string, pickup velocity | 2 s mono wav, 48 kHz, float32 |
03370d1Register solution repositories and build the comparison site here; optional HDF5 field export in the schema and specJeremy Magland 11Each challenge lives in `challenges/<challenge-id>/` and consists of a `README.md`, which is the authoritative specification, and a `challenge.json`, which repeats the parameters, the output format, the error measures, and the recommended grid for optional field exports in machine-readable form. A challenge is never changed once solutions exist for it; a corrected or modified model gets a new ID (for example, a `-v2` suffix).
13## How a challenge is specified
15The README states the governing equations, boundary and initial conditions, and all parameter values in SI units, and defines the output as a function of continuous time. It then states exactly how that function is sampled into the deliverable file. The aim is that the true solution is a well-defined function that does not depend on any discretization, so that methods of very different kinds (modal, finite difference, finite element, integral equation) can be compared on equal terms.
03370d1Register solution repositories and build the comparison site here; optional HDF5 field export in the schema and specJeremy Magland 17## Solution repositories and the registry
1aa0831Use the shared challenge-tools package; describe the field grid by axes and shapeJeremy Magland 19Solutions are not stored here. A solution repository is any public repository on this vault containing `solutions/<challenge-id>/<submission-id>.json` files, validated against the [submission schema](https://vault1.magland.org/challenges/challenge-tools/blob/main/schema.json), with the output files beside them or at URLs. One JSON file holds one or more solutions from one submitter for one challenge. Each solution names its wav, and may report a method name, discretization parameters, self-reported wall-clock time, hardware, a link to the source code, a link to a visualization, and an optional export of the solution field for visualization. What else a solution repository contains, such as solver code or its own visualization site, is up to its authors. [challenges/string-solutions](https://vault1.magland.org/challenges/string-solutions) is the first one and serves as an example.
03370d1Register solution repositories and build the comparison site here; optional HDF5 field export in the schema and specJeremy Magland 20
21The repositories that take part in the comparison are listed in [registry/solution-repos.json](registry/solution-repos.json), one entry per repository with its branch and the path of its `solutions` directory. To register a repository, open a pull request adding an entry. The comparison workflow clones every registered repository at the head of its branch, so a push to a solution repository is picked up the next time the workflow runs: on any push to this repository, or when it is dispatched by hand (`mochi workflow run .mochi/workflows/results.yml --repo challenges/string`). Nothing in the vault triggers across repositories.
23## How solutions are scored
25There is no externally provided ground truth. Every solution is compared with a reference solution chosen from among the submissions, using the error measures defined in the challenge README. All measures are reported; one of them is designated primary and is used for ranking.
c183eeaBreak reference ties by convergence gap; document the rule's limitationsJeremy Magland 27The reference is chosen as follows. Within a submission, the submitter may assert that some solutions are more accurate than others through the field `more_accurate_than` (a convergence series at increasing resolution is the typical case). Only orderings within a single submission are trusted. The candidates for reference are the maximal elements of each submission's ordering. A candidate is scored by the number of asserted pairs from other submissions whose ordering is reproduced by the primary error to that candidate; the candidate reproducing the largest fraction wins. Ties are broken first by the candidate's convergence gap, that is, the error between it and the nearest solution of its own submission asserted to be less accurate (a sequence whose last two members agree closely is claiming a tighter accuracy), and then by the smallest mean error to the other candidates. When no orderings are asserted at all, this rule reduces to choosing the candidate nearest the others. A challenge may override the automatic choice by naming a solution in `reference_solution` in its `challenge.json`.
29The rule has a clear limitation: it is only as good as the asserted orderings. A submission whose sequence is not in fact monotone (for example, a scheme with two error terms of opposite sign that cancel at one resolution) makes a false assertion, and a candidate that happens to reproduce that false pair can be preferred over a more accurate one. With only two submissions this cannot be detected automatically. The results page therefore shows, for every candidate, which asserted pairs it fails to reproduce, so that a reader can judge the choice, and the maintainers can override it with `reference_solution`. More independent submissions make the rule more reliable.
03370d1Register solution repositories and build the comparison site here; optional HDF5 field export in the schema and specJeremy Magland 31The results page lists every solution with its errors relative to the reference, its reported time and hardware, an audio player, and links to the submission file, its repository, the source code, the exported fields, and the submitter's visualization. It also reports which asserted orderings the chosen reference does or does not reproduce, and the full matrix of pairwise errors. Submissions are identified as `<repository>/<submission-id>` so that identifiers from different repositories cannot collide.
33## Tools
1aa0831Use the shared challenge-tools package; describe the field grid by axes and shapeJeremy Magland 35Validation, scoring, reference selection, and the comparison site are implemented once, in [challenges/challenge-tools](https://vault1.magland.org/challenges/challenge-tools), and shared by every challenge repository. The workflow in `.mochi/workflows/results.yml` installs that package, fetches the registered solution repositories, and builds the site:
37```
1aa0831Use the shared challenge-tools package; describe the field grid by axes and shapeJeremy Magland 38pip install git+https://vault1.magland.org/challenges/challenge-tools
03370d1Register solution repositories and build the comparison site here; optional HDF5 field export in the schema and specJeremy Magland 39challenge-tools fetch --registry registry/solution-repos.json --clones clones
40challenge-tools build --challenges challenges --registry registry/solution-repos.json --clones clones --out site
1aa0831Use the shared challenge-tools package; describe the field grid by axes and shapeJeremy Magland 43A single local directory can stand in for the registry during development with `--solutions <dir>` in place of `--registry` and `--clones`. The challenge README is the specification and the code is intended to follow it exactly.