Break reference ties by convergence gap; document the rule's limitations
4 changed files+30−5
README.mdmodified+1−1View file
@@ -22,7 +22,7 @@ A submission is a JSON file in the solutions repository at `solutions/<challenge
2222
2323 There is no externally provided ground truth. Every solution is compared with a reference solution chosen from among the submissions, using the error measures defined in the challenge README. All measures are reported; one of them is designated primary and is used for ranking.
2424
25-The reference is chosen as follows. Within a submission, the submitter may assert that some solutions are more accurate than others through the field `more_accurate_than` (a convergence series at increasing resolution is the typical case). Only orderings within a single submission are trusted. The candidates for reference are the maximal elements of each submission's ordering. A candidate is scored by the number of asserted pairs from other submissions whose ordering is reproduced by the primary error to that candidate; the candidate reproducing the largest fraction wins, and ties are broken by the smallest mean error to the other candidates. When no orderings are asserted at all, this rule reduces to choosing the candidate nearest the others. A challenge may override the automatic choice by naming a solution in `reference_solution` in its `challenge.json`.
25+The reference is chosen as follows. Within a submission, the submitter may assert that some solutions are more accurate than others through the field `more_accurate_than` (a convergence series at increasing resolution is the typical case). Only orderings within a single submission are trusted. The candidates for reference are the maximal elements of each submission's ordering. A candidate is scored by the number of asserted pairs from other submissions whose ordering is reproduced by the primary error to that candidate; the candidate reproducing the largest fraction wins. Ties are broken first by the candidate's convergence gap, that is, the error between it and the nearest solution of its own submission asserted to be less accurate (a sequence whose last two members agree closely is claiming a tighter accuracy), and then by the smallest mean error to the other candidates. When no orderings are asserted at all, this rule reduces to choosing the candidate nearest the others. A challenge may override the automatic choice by naming a solution in `reference_solution` in its `challenge.json`.
2626
2727 The rule has a clear limitation: it is only as good as the asserted orderings. A submission whose sequence is not in fact monotone (for example, a scheme with two error terms of opposite sign that cancel at one resolution) makes a false assertion, and a candidate that happens to reproduce that false pair can be preferred over a more accurate one. With only two submissions this cannot be detected automatically. The results page therefore shows, for every candidate, which asserted pairs it fails to reproduce, so that a reader can judge the choice, and the maintainers can override it with `reference_solution`. More independent submissions make the rule more reliable.
2828
tests/test_tools.pymodified+21−0View file
@@ -54,3 +54,24 @@ def test_reference_selection_medoid_fallback():
5454 err = {a: {b: abs(pos[a] - pos[b]) for b in keys} for a in keys}
5555 rep = select_reference(info, err)
5656 assert rep["reference"] == "q/b"
57+
58+
59+def test_reference_selection_tie_broken_by_convergence_gap():
60+ subs = [
61+ {"submission_id": "x", "solutions": [
62+ {"solution_id": "a", "more_accurate_than": []},
63+ {"solution_id": "b", "more_accurate_than": ["a"]},
64+ ]},
65+ {"submission_id": "y", "solutions": [
66+ {"solution_id": "c", "more_accurate_than": []},
67+ {"solution_id": "d", "more_accurate_than": ["c"]},
68+ ]},
69+ ]
70+ info = collect_order(subs)
71+ keys = ["x/a", "x/b", "y/c", "y/d"]
72+ # both chains monotone toward 0; x/b is much closer to its sibling than y/d is to its own
73+ truth = {"x/a": 0.03, "x/b": 0.0001, "y/c": 1.0, "y/d": 0.01}
74+ err = {a: {b: abs(truth[a] - truth[b]) for b in keys} for a in keys}
75+ rep = select_reference(info, err)
76+ assert [c["score"] for c in rep["candidates"]] == [1.0, 1.0]
77+ assert rep["reference"] == "x/b"
tools/challenge_tools/build.pymodified+3−3View file
@@ -161,7 +161,7 @@ def render_challenge_page(res: dict, challenge: dict, links: dict) -> str:
161161 sel = res["reference_selection"]
162162 cand_rows = "".join(
163163 f"<tr><td>{html.escape(c['candidate'])}</td><td class='num'>{c['pairs_preserved']}/{c['pairs_total']}</td>"
164- f"<td class='num'>{_fmt(c['mean_error_to_other_candidates'])}</td>"
164+ f"<td class='num'>{_fmt(c['convergence_gap'])}</td><td class='num'>{_fmt(c['mean_error_to_other_candidates'])}</td>"
165165 f"<td>{html.escape(', '.join(f'{w} vs {b}' for w, b in c['violated_pairs']))}</td></tr>"
166166 for c in sel.get("candidates", [])
167167 )
@@ -197,8 +197,8 @@ def render_challenge_page(res: dict, challenge: dict, links: dict) -> str:
197197 {"".join(body)}
198198 </table></div>
199199 <details><summary>Reference selection</summary>
200-<p>Candidates are the maximal elements of each submission's asserted order. The score is the number of asserted pairs from other submissions whose order is preserved by the primary error to this candidate. Ties are broken by the mean primary error to the other candidates.</p>
201-<div class="wrap"><table><tr><th>Candidate</th><th>Pairs preserved</th><th>Mean error to other candidates</th><th>Violated pairs</th></tr>{cand_rows}</table></div>
200+<p>Candidates are the maximal elements of each submission's asserted order. The score is the number of asserted pairs from other submissions whose order is preserved by the primary error to this candidate. Ties are broken first by the convergence gap (the primary error between the candidate and the nearest solution of its own submission asserted to be less accurate; smaller means a tighter convergence claim), then by the mean primary error to the other candidates.</p>
201+<div class="wrap"><table><tr><th>Candidate</th><th>Pairs preserved</th><th>Convergence gap</th><th>Mean error to other candidates</th><th>Violated pairs</th></tr>{cand_rows}</table></div>
202202 <h4>Order check against the chosen reference</h4>{viol_html}
203203 </details>
204204 <details><summary>Pairwise errors (row vs column as reference)</summary>{"".join(pair_tables)}</details>
tools/challenge_tools/reference.pymodified+5−1View file
@@ -60,12 +60,15 @@ def select_reference(
6060 violated.append([worse, better])
6161 others = [c for c in candidates if c != cand]
6262 mean_err = sum(err[c][cand] for c in others) / len(others) if others else 0.0
63+ siblings = [worse for worse, better, _sid in info.pairs if better == cand]
64+ gap = min((err[w][cand] for w in siblings), default=None)
6365 rows.append(
6466 {
6567 "candidate": cand,
6668 "pairs_total": len(external),
6769 "pairs_preserved": preserved,
6870 "score": (preserved / len(external)) if external else None,
71+ "convergence_gap": gap,
6972 "mean_error_to_other_candidates": mean_err,
7073 "violated_pairs": violated,
7174 }
@@ -73,7 +76,8 @@ def select_reference(
7376
7477 def sort_key(row):
7578 score = row["score"] if row["score"] is not None else -1.0
76- return (-score, row["mean_error_to_other_candidates"])
79+ gap = row["convergence_gap"] if row["convergence_gap"] is not None else float("inf")
80+ return (-score, gap, row["mean_error_to_other_candidates"])
7781
7882 rows.sort(key=sort_key)
7983 return {