1# Instrument simulation challenges: string (tier 0)
3This repository defines simulation challenges for a damped string. It is the first, and simplest, repository in a collection of challenges for physics-based simulation of musical instruments, in which each challenge is a fully specified continuum model together with a precisely defined output signal and error measures. Solutions are not stored here; they are submitted to the companion repository [challenges/string-solutions](https://vault1.magland.org/challenges/string-solutions), and the results table is published at [challenges/string-solutions/site](https://vault1.magland.org/challenges/string-solutions/site/).
5## Challenges
7| ID | Title | Deliverable |
8| --- | --- | --- |
9| [string-pluck-v1](challenges/string-pluck-v1/README.md) | Plucked damped string, pickup velocity | 2 s mono wav, 48 kHz, float32 |
11Each challenge lives in `challenges/<challenge-id>/` and consists of a `README.md`, which is the authoritative specification, and a `challenge.json`, which repeats the parameters, the output format, and the error measures in machine-readable form. A challenge is never changed once solutions exist for it; a corrected or modified model gets a new ID (for example, a `-v2` suffix).
13## How a challenge is specified
15The README states the governing equations, boundary and initial conditions, and all parameter values in SI units, and defines the output as a function of continuous time. It then states exactly how that function is sampled into the deliverable file. The aim is that the true solution is a well-defined function that does not depend on any discretization, so that methods of very different kinds (modal, finite difference, finite element, integral equation) can be compared on equal terms.
17## How solutions are submitted
19A submission is a JSON file in the solutions repository at `solutions/<challenge-id>/<submission-id>.json`, validated against [schema/solution.schema.json](schema/solution.schema.json). One file holds one or more solutions from one submitter for one challenge. Each solution names its output file (a path relative to the JSON file, or a URL), and may report a method name, discretization parameters, self-reported wall-clock time, hardware, a link to the source code, and a link to a visualization. The submission is made by pull request (or by direct push, for collaborators) to the solutions repository. See that repository's README for the details.
21## How solutions are scored
23There is no externally provided ground truth. Every solution is compared with a reference solution chosen from among the submissions, using the error measures defined in the challenge README. All measures are reported; one of them is designated primary and is used for ranking.
25The reference is chosen as follows. Within a submission, the submitter may assert that some solutions are more accurate than others through the field `more_accurate_than` (a convergence series at increasing resolution is the typical case). Only orderings within a single submission are trusted. The candidates for reference are the maximal elements of each submission's ordering. A candidate is scored by the number of asserted pairs from other submissions whose ordering is reproduced by the primary error to that candidate; the candidate reproducing the largest fraction wins. Ties are broken first by the candidate's convergence gap, that is, the error between it and the nearest solution of its own submission asserted to be less accurate (a sequence whose last two members agree closely is claiming a tighter accuracy), and then by the smallest mean error to the other candidates. When no orderings are asserted at all, this rule reduces to choosing the candidate nearest the others. A challenge may override the automatic choice by naming a solution in `reference_solution` in its `challenge.json`.
27The rule has a clear limitation: it is only as good as the asserted orderings. A submission whose sequence is not in fact monotone (for example, a scheme with two error terms of opposite sign that cancel at one resolution) makes a false assertion, and a candidate that happens to reproduce that false pair can be preferred over a more accurate one. With only two submissions this cannot be detected automatically. The results page therefore shows, for every candidate, which asserted pairs it fails to reproduce, so that a reader can judge the choice, and the maintainers can override it with `reference_solution`. More independent submissions make the rule more reliable.
29The results page lists every solution with its errors relative to the reference, its reported time and hardware, an audio player, and links to code and visualizations. It also reports which asserted orderings the chosen reference does or does not reproduce, and the full matrix of pairwise errors.
31## Tools
33The `tools/` directory holds a small Python package, `challenge_tools`, that validates submissions, computes the error measures, selects the reference, and writes the static results site. The solutions repository runs it in a workflow on every push.
35```
36pip install .
37challenge-tools validate --challenges challenges --solutions ../string-solutions/solutions
38challenge-tools build --challenges challenges --solutions ../string-solutions/solutions --out site
39```
41The error measures are implemented in `tools/challenge_tools/metrics.py` and the reference rule in `tools/challenge_tools/reference.py`; the challenge README is the specification and the code is intended to follow it exactly.