6292f12A benchmark chapter, and offline benchmarks to compare it againstJeremy Magland 1# Offline benchmarks
3The benchmark chapter of the tutorial runs its four methods in the browser
4through numbl. The scripts here run the two best of those methods, the
5harmonic basis and the graded-panel double layer, outside the browser, on the
1a8bb6aMake the cross-language benchmark measure the languagesJeremy Magland 6same Pac-Man problem and the same error lattice.
8Run whichever of these your machine has, from the repository root:
10```
1a8bb6aMake the cross-language benchmark measure the languagesJeremy Magland 11python3 bench/python/bench_pacman.py # needs NumPy and SciPy
6292f12A benchmark chapter, and offline benchmarks to compare it againstJeremy Magland 12julia bench/julia/bench_pacman.jl # standard library only
13matlab -batch "run('bench/matlab/bench_pacman.m')"
14```
16Each writes a `bench-pacman-*.json` file (the MATLAB one lands in the
17repository root, the others in the current directory). Load those files from
18the "load offline results" button under the benchmark chapter's
19work-precision plot, and they appear as dashed curves beside the browser's.
21The point of running the scripts yourself, rather than reading times measured
22on someone else's machine, is that your browser results and your offline
23results then share a clock and can honestly share an axis.
25The MATLAB harness runs the chapter's own driver and editable script,
26unchanged, in real MATLAB. The Python and Julia scripts are ports; the check
27that they are faithful is that their error columns land on top of the
1a8bb6aMake the cross-language benchmark measure the languagesJeremy Magland 28browser's curves, since the error depends only on the mathematics. On the
29double layer the three environments agree bit for bit; on the basis method
30they agree to within a factor of two at the $10^{-15}$ roundoff floor, where
31the order of the QR pivoting is all that separates them.
33## Comparing languages, not authors
35A cross-language benchmark measures how carefully each port was written unless
36that is deliberately controlled for, so three rules are held to here. The
37chapter's prose discusses them; in the code they look like this.
39**Each port uses the fastest shape its own language offers, and they differ.**
40The kernel matrix is assembled by a scalar double loop in the MATLAB script
41(hence in numbl too) and in Julia, because that is about twice as fast as
42implicit expansion in MATLAB and numbl and much faster than broadcasting in
43Julia. NumPy cannot run that loop at any acceptable speed, so the Python port
44assembles with whole-array expressions, in place: the obvious version, one
45broadcast statement per line, needs five n-by-n temporaries and is about 1.6
46times slower for that reason alone. Insisting on one shape everywhere would
47make the benchmark less fair, not more.
49**The same linear algebra is requested everywhere.** `numpy.linalg.lstsq` is
50an SVD, while MATLAB's and Julia's backslash are pivoted QR on a rectangular
51system, so the Python port calls SciPy's QR driver (`gelsy`) instead. For the
52square systems all three call the same LAPACK and land within ten per cent.
54**One timing protocol, everywhere, including in the page.** Keep the fastest
55of several runs: at least three whenever a solve is cheap enough to allow it,
56many more when it is very cheap, one when it is expensive. A single
57measurement of a fifty-millisecond solve varies by a factor of two with
58whatever else the machine is doing.
60Each label records the BLAS thread count, since that is a real difference
61between environments and otherwise an invisible one: on the machine these
62were developed on, MATLAB takes ten threads, NumPy twelve, and Julia six.
64## The file format
66```json
67{
68 "schema": "laplace-dirichlet-bench-1",
69 "label": "what the legend should call this run",
70 "kind": "both",
71 "date": "2026-08-20",
72 "methods": [
73 {
74 "name": "harmonic basis, least squares",
75 "slot": "mps",
76 "ns": [4, 8, 16],
77 "E": [8.1e-4, 6.3e-6, 3.6e-10],
78 "T": [0.2, 0.4, 0.9]
79 }
80 ]
81}
82```
84`slot` picks the plot colour and marker and is one of `fd2`, `fd4`,
85`spectral`, `fast`, `iterative`, `mps`, `bie`; anything else falls back to
86`fd2`. `E` is the maximum absolute error over the chapter's fixed interior
87lattice, `T` the solve time in milliseconds, and entries may be `null` for a
88resolution a method skipped. `kind` records which ground truth was measured
89(`both`, `sing`, or `smooth`); the page warns if it differs from the one the
90lab is currently showing. Nothing stops you from writing a file in this
91format from any other language or machine; the legend will say whatever
92`label` says.