9d60edeOverview of the challenges collectionJeremy Magland 1# Physics-based simulation of musical instruments: the challenges collection
3This collection holds a set of simulation challenges for the sound of struck and plucked instruments, together with the solutions people have submitted and the machinery that compares them. This repository is the entry point. It explains why the collection exists, how it is organized, how a solution is scored, and how to take part. The other repositories in the collection are listed at the end.
5The results are published as static sites, one per tier:
7| Tier | Challenges | Comparison of solutions |
8| --- | --- | --- |
9| 0, string | [challenges/string](https://vault1.magland.org/challenges/string) | [results](https://vault1.magland.org/challenges/string/site/) |
10| 1, plate | [challenges/plate](https://vault1.magland.org/challenges/plate) | [results](https://vault1.magland.org/challenges/plate/site/) |
90c690fAdd the cymbal tierJeremy Magland 11| 2, cymbal | [challenges/cymbal](https://vault1.magland.org/challenges/cymbal) | [results](https://vault1.magland.org/challenges/cymbal/site/) |
9d60edeOverview of the challenges collectionJeremy Magland 12
13## Background
1b04eceReword the opening of the background sectionJeremy Magland 15The motivating application for this project is realistic instrument sound generated from physical models. Producing it is not trivial. It requires coupled elastodynamics and acoustics, stiff time integration, and careful trade-offs between accuracy and cost, and the ear judges the result harshly: it detects errors in pitch, decay, and timbre that are easy to miss in a convergence plot. The primary contribution we are after is the numerical methods themselves and benchmarks of their accuracy against cost. The same physics arises in vibroacoustics and acoustic scattering, so the methods should be relevant beyond music.
9d60edeOverview of the challenges collectionJeremy Magland 16
17Simulation methods are hard to compare fairly unless they are applied to the same problem with the same reference and the same error measures. A challenge here is therefore a fully specified continuum model together with a precisely defined output signal and error measures. The specification states the governing equations, boundary and initial conditions, every parameter in SI units, and exactly how the output is sampled into a file. The intent is that the true solution is a well-defined function that does not depend on any discretization, so that methods of very different kinds, modal, finite difference, finite element, integral equation, can be compared on equal terms. Tiers add realism: toy problems first, then thin plates, then geometric nonlinearity, then radiation into air, up to complete instruments.
19There is no externally supplied ground truth. The reference for each challenge is chosen from among the submitted solutions by a rule described below, and the central question a challenge answers is how much computation a method can save while remaining close to the most accurate solution available.
21## How the collection is organized
23Each tier is a pair of repositories, plus shared tooling.
90c690fAdd the cymbal tierJeremy Magland 25- **A challenge repository** (`challenges/string`, `challenges/plate`, `challenges/cymbal`) holds the specifications: one directory per challenge with a `README.md`, which is the authoritative definition, and a `challenge.json`, which repeats the parameters, output format, error measures, and the grid of the optional field export in machine-readable form. It also holds a registry of solution repositories and the workflow that builds the comparison site.
26- **Solution repositories** (`challenges/string-solutions`, `challenges/plate-solutions`, `challenges/cymbal-solutions`, and any others that are registered) hold submissions: a JSON file per submission with metadata and one or more solutions, the output wav files, optional field exports for visualization, and whatever else the authors want, such as solver code or their own visualization site. A solution repository runs only the validator; it makes no comparisons of its own.
9d60edeOverview of the challenges collectionJeremy Magland 27- **The tools** (`challenges/challenge-tools`) are a small Python package with the submission schema, the validator, the error measures, the reference-selection rule, and the generator of the comparison site. Every challenge repository installs it in its workflow.
29The comparison workflow of a challenge repository clones every registered solution repository, reads the committed output files, computes every pairwise error, chooses the reference, and deploys the site. It runs on every push to the challenge repository or when dispatched by hand. Nothing in the vault triggers across repositories, so a push to a solution repository is picked up at the next run of the challenge repository's workflow. All solvers run on the submitters' own machines; the workflow runner only scores committed files.
31## Submissions
33A submission is a JSON file at `solutions/<challenge-id>/<submission-id>.json` in a solution repository, validated against the [schema](https://vault1.magland.org/challenges/challenge-tools/blob/main/schema.json). It names the submitter and holds one or more solutions. Each solution points to its output file (a path beside the JSON, or a URL), and may report the method, its discretization parameters, self-reported wall-clock time and hardware, a link to the source code, a link to a visualization, and an optional field export. Submissions are identified as `<repository>/<submission-id>` in the results, so identifiers from different repositories cannot collide.
35A submission may assert an accuracy ordering among its own solutions through `more_accurate_than`; a sequence at increasing resolution is the typical case. These assertions are what allow the reference to be chosen automatically, so a submission that includes a convergence series is worth more to the collection than a single solution.
37Output files are exact, unnormalized signals in a fixed physical scale stated by each challenge, as 32-bit float wav. They have a large crest factor (a loud strike followed by a much quieter ring) and sound faint when played raw; the results sites play them through a gain and a limiter, and each challenge suggests a playback gain. The files themselves are never altered.
39## Scoring and the reference
41Every solution is compared with the reference under the error measures defined in the challenge, at present the relative waveform error, which is strict and penalizes phase drift, and a relative spectrogram error, which is closer to what a listener notices. All measures are reported and one, named in the challenge, ranks.
43The reference is chosen as follows. Only orderings asserted within a single submission are trusted. The candidates are the maximal elements of each submission's ordering. A candidate is scored by the fraction of asserted pairs from other submissions whose order is reproduced by the primary error to that candidate; the candidate reproducing the largest fraction wins. Ties are broken first by the candidate's convergence gap, the error between it and the nearest solution of its own submission asserted to be less accurate, and then by the smallest mean error to the other candidates. When nothing is asserted, the rule reduces to picking the candidate nearest the others.
45The rule is only as good as the assertions. A sequence that is not in fact monotone makes a false claim, and with few submissions a candidate that happens to agree with that claim can be preferred over a more accurate one. The results pages therefore show, for every candidate, which asserted pairs it fails to reproduce, so that a reader can judge the choice, and a challenge may override the automatic choice by naming a reference explicitly. More independent submissions make the rule more reliable, and this is the main reason to submit even a modest method.
47## The challenges so far
49| Challenge | Model | Output |
50| --- | --- | --- |
51| [string-pluck-v1](https://vault1.magland.org/challenges/string/blob/main/challenges/string-pluck-v1/README.md) | Damped string with fixed ends, released from its static equilibrium under a finger load | 2 s of pickup velocity |
52| [plate-strike-v1](https://vault1.magland.org/challenges/plate/blob/main/challenges/plate-strike-v1/README.md) | Free 12 in aluminum plate, linear 3D elastodynamics with Kelvin-Voigt damping, struck by a short force pulse | 3 s of radiated pressure by the Rayleigh integral |
53| [plate-strike-nl-v1](https://vault1.magland.org/challenges/plate/blob/main/challenges/plate-strike-nl-v1/README.md) | The same plate as a Saint Venant-Kirchhoff solid (geometrically nonlinear), struck by a soft mallet | 3 s of radiated pressure |
90c690fAdd the cymbal tierJeremy Magland 54| [cymbal-flat-disk-v1](https://vault1.magland.org/challenges/cymbal/blob/main/challenges/cymbal-flat-disk-v1/README.md) | Flat, free 16 in bronze disk of uniform thickness, Saint Venant-Kirchhoff, struck near the rim by a stick-like pulse | 6 s of radiated pressure |
cbd127bAdd cymbal-bow-v1 to the tableJeremy Magland 55| [cymbal-bow-v1](https://vault1.magland.org/challenges/cymbal/blob/main/challenges/cymbal-bow-v1/README.md) | The same disk bowed into a shallow paraboloid (25 mm rise): the effect of curvature alone | 6 s of radiated pressure |
e4c25d0Add cymbal-bell-v1 to the tableJeremy Magland 56| [cymbal-bell-v1](https://vault1.magland.org/challenges/cymbal/blob/main/challenges/cymbal-bell-v1/README.md) | The bowed disk with a cosine-dome bell and a mounting hole: the effect of the bell alone | 6 s of radiated pressure |
bee2ceaAdd cymbal-hammered-v1 to the tableJeremy Magland 57| [cymbal-hammered-v1](https://vault1.magland.org/challenges/cymbal/blob/main/challenges/cymbal-hammered-v1/README.md) | The belled disk with twenty hammer dents at fixed positions: the effect of breaking the symmetry | 6 s of radiated pressure |
3740f96Add cymbal-tapered-v1 to the tableJeremy Magland 58| [cymbal-tapered-v1](https://vault1.magland.org/challenges/cymbal/blob/main/challenges/cymbal-tapered-v1/README.md) | The hammered cymbal lathed to a taper (1.2 to 0.7 mm): the effect of the thickness profile | 6 s of radiated pressure |
9d60edeOverview of the challenges collectionJeremy Magland 59
90c690fAdd the cymbal tierJeremy Magland 60The string challenge is a hello world: its modal series solution is essentially exact, so it exercises the submission and scoring pipeline and gives a check on the reference rule. The plate challenges define three-dimensional models, so the thin-plate solutions submitted so far (Kirchhoff-Love finite elements for the linear plate, a modal von Kármán method for the nonlinear one) are approximations whose error relative to a future converged three-dimensional solution includes the modeling error of plate theory. Until such a solution arrives, the reference is the best thin-plate solution and the errors measure only differences within plate theory; the results pages say so. The cymbal tier starts from a flat disk and will add the features of a real cymbal one at a time (bell, bow, hammering, varying thickness), each as a new challenge, so that the effect of each on the sound and on the cost of computing it can be seen in isolation; its first solutions are von Kármán modal solutions on Cartesian grids that approximate the disk by a staircase.
9d60edeOverview of the challenges collectionJeremy Magland 61
62## Conventions
9cde185Conventions: free bodies and the linearized rigid motionJeremy Magland 64- **Free bodies.** The plates and disks are free, and the strikes impart momentum and angular momentum. The outputs are defined with the rigid motion linearized: the acceleration relative to the best-fit rigid frame plus that frame's translational and angular acceleration, which keeps the kick of the strike in the sound and leaves out the slow turning of the body that only an exact three-dimensional solution would show. For modal thin-plate solvers this is what they compute with their rigid modes included.
9d60edeOverview of the challenges collectionJeremy Magland 65- A challenge is never changed once solutions from others exist for it. A corrected or modified model gets a new identifier, for example a `-v2` suffix.
66- Identifiers (challenges, submissions, solutions) are lowercase letters, digits, and hyphens.
67- Field exports for visualization are HDF5 files on a grid recommended by the challenge; they are not scored. Files of a few megabytes live in plain git, larger ones in git LFS or at an external URL.
68- Timing is self-reported wall-clock time for producing the output signal, on the hardware named in the submission.
70## Taking part
72- **Submit a solution.** Add a submission to an existing solution repository by pull request, or start a repository of your own with the same `solutions/` layout and register it by a pull request to the challenge repository's `registry/solution-repos.json`. The solution repositories' READMEs give the steps, and the validator can be run locally with `pip install git+https://vault1.magland.org/challenges/challenge-tools`.
73- **Propose a challenge.** Open an issue or a pull request on the relevant challenge repository with a specification in the same form: continuum model, parameters, output, error measures. New tiers get a repository pair of their own.
74- **Improve the tools.** The error measures and the reference rule are implemented in `challenge-tools`; the challenge READMEs are the specification and the code is intended to follow them exactly.
76## Roadmap
90c690fAdd the cymbal tierJeremy Magland 78The tiers sketched in the research plan, for a cymbal as the running example, are: a flat plate struck at a point (tier 1, done in linear and nonlinear form); a curved, cymbal-shaped shell (tier 2, begun with the flat disk); large-amplitude strikes with geometric nonlinearity on that shell (tier 3); and sound radiated into the surrounding air by a coupled structure and acoustic simulation (tier 4), replacing the Rayleigh integral used so far. Alongside the tiers we intend to pursue simulations of complete instruments, starting with a cymbal on a stand, which are harder to benchmark but show which effects matter most for the sound and so guide which challenges to add. Open questions include which errors are audible, since a cheaper model can have a large waveform error yet sound identical, and the collection is a natural place to host listening tests once there are enough solutions to compare.
9d60edeOverview of the challenges collectionJeremy Magland 79
80## Repositories in this collection
82| Repository | Contents |
83| --- | --- |
84| [overview](https://vault1.magland.org/challenges/overview) | this document |
85| [challenge-tools](https://vault1.magland.org/challenges/challenge-tools) | schema, validator, error measures, reference rule, site generator |
86| [string](https://vault1.magland.org/challenges/string) | tier 0 specifications, registry, comparison site |
87| [string-solutions](https://vault1.magland.org/challenges/string-solutions) | solutions to the string challenges, with a 2D field viewer |
88| [plate](https://vault1.magland.org/challenges/plate) | tier 1 specifications, registry, comparison site |
89| [plate-solutions](https://vault1.magland.org/challenges/plate-solutions) | solutions to the plate challenges, with a 3D field viewer |
90c690fAdd the cymbal tierJeremy Magland 90| [cymbal](https://vault1.magland.org/challenges/cymbal) | tier 2 specifications, registry, comparison site |
91| [cymbal-solutions](https://vault1.magland.org/challenges/cymbal-solutions) | solutions to the cymbal challenges, with a 3D field viewer |