Instrument simulation challenges: string (tier 0)#

This repository defines simulation challenges for a damped string, and publishes the comparison of every registered solution. It is the first, and simplest, repository in a collection of challenges for physics-based simulation of musical instruments, in which each challenge is a fully specified continuum model together with a precisely defined output signal and error measures. Solutions live in separate repositories that are registered here; the comparison is at challenges/string/site.

Challenges#

Each challenge lives in challenges/<challenge-id>/ and consists of a README.md, which is the authoritative specification, and a challenge.json, which repeats the parameters, the output format, the error measures, and the recommended grid for optional field exports in machine-readable form. A challenge is never changed once solutions exist for it; a corrected or modified model gets a new ID (for example, a -v2 suffix).

How a challenge is specified#

The README states the governing equations, boundary and initial conditions, and all parameter values in SI units, and defines the output as a function of continuous time. It then states exactly how that function is sampled into the deliverable file. The aim is that the true solution is a well-defined function that does not depend on any discretization, so that methods of very different kinds (modal, finite difference, finite element, integral equation) can be compared on equal terms.

Solution repositories and the registry#

Solutions are not stored here. A solution repository is any public repository on this vault containing solutions/<challenge-id>/<submission-id>.json files, validated against schema/solution.schema.json, with the output files beside them or at URLs. One JSON file holds one or more solutions from one submitter for one challenge. Each solution names its wav, and may report a method name, discretization parameters, self-reported wall-clock time, hardware, a link to the source code, a link to a visualization, and an optional export of the solution field for visualization. What else a solution repository contains, such as solver code or its own visualization site, is up to its authors. challenges/string-solutions is the first one and serves as an example.

The repositories that take part in the comparison are listed in registry/solution-repos.json, one entry per repository with its branch and the path of its solutions directory. To register a repository, open a pull request adding an entry. The comparison workflow clones every registered repository at the head of its branch, so a push to a solution repository is picked up the next time the workflow runs: on any push to this repository, or when it is dispatched by hand (mochi workflow run .mochi/workflows/results.yml --repo challenges/string). Nothing in the vault triggers across repositories.

How solutions are scored#

There is no externally provided ground truth. Every solution is compared with a reference solution chosen from among the submissions, using the error measures defined in the challenge README. All measures are reported; one of them is designated primary and is used for ranking.

The reference is chosen as follows. Within a submission, the submitter may assert that some solutions are more accurate than others through the field more_accurate_than (a convergence series at increasing resolution is the typical case). Only orderings within a single submission are trusted. The candidates for reference are the maximal elements of each submission's ordering. A candidate is scored by the number of asserted pairs from other submissions whose ordering is reproduced by the primary error to that candidate; the candidate reproducing the largest fraction wins. Ties are broken first by the candidate's convergence gap, that is, the error between it and the nearest solution of its own submission asserted to be less accurate (a sequence whose last two members agree closely is claiming a tighter accuracy), and then by the smallest mean error to the other candidates. When no orderings are asserted at all, this rule reduces to choosing the candidate nearest the others. A challenge may override the automatic choice by naming a solution in reference_solution in its challenge.json.

The rule has a clear limitation: it is only as good as the asserted orderings. A submission whose sequence is not in fact monotone (for example, a scheme with two error terms of opposite sign that cancel at one resolution) makes a false assertion, and a candidate that happens to reproduce that false pair can be preferred over a more accurate one. With only two submissions this cannot be detected automatically. The results page therefore shows, for every candidate, which asserted pairs it fails to reproduce, so that a reader can judge the choice, and the maintainers can override it with reference_solution. More independent submissions make the rule more reliable.

The results page lists every solution with its errors relative to the reference, its reported time and hardware, an audio player, and links to the submission file, its repository, the source code, the exported fields, and the submitter's visualization. It also reports which asserted orderings the chosen reference does or does not reproduce, and the full matrix of pairwise errors. Submissions are identified as <repository>/<submission-id> so that identifiers from different repositories cannot collide.

Tools#

The tools/ directory holds a Python package, challenge_tools, that fetches the registered repositories, validates submissions, computes the error measures, selects the reference, and writes the static comparison site. The workflow in .mochi/workflows/results.yml runs it on every push and on dispatch. Solution repositories are expected to run only the validator.

pip install .
challenge-tools fetch --registry registry/solution-repos.json --clones clones
challenge-tools validate --challenges challenges --registry registry/solution-repos.json --clones clones
challenge-tools build --challenges challenges --registry registry/solution-repos.json --clones clones --out site

A single local directory can stand in for the registry during development with --solutions <dir> in place of --registry and --clones. The error measures are implemented in tools/challenge_tools/metrics.py and the reference rule in tools/challenge_tools/reference.py; the challenge README is the specification and the code is intended to follow it exactly.