3Tips for future agents working in this repo. It combines the p2p techniques of
4the sibling projects `commonview` (auto-connecting mesh) and `commoncall`
5(WebRTC media, settings, screen share) — read those first; this file only
6covers what is different here.
8## Architecture
10```
11src/p2p/
12 identity.ts schnorr keypair; pubkey hex = peer ID — ported from commoncall
13 nostr.ts minimal relay client + topic scheme — ported (roomTopic takes a room ID)
14 peer.ts WebRTC wrapper: media + control channel — ported (replaceTrack generalized to audio|video)
15 settings.ts shared ROOM settings, quality presets — default quality is 'medium', not 'auto'
06b5f04Optional TURN relay, gated by a token and shared across the roomJeremy Magland 16 turn.ts optional TURN: build-time endpoint, credential fetch, sanitizers
17 network.ts the heart: rooms, presence, mesh, media, settings sync, relay sharing
8107f0eOptional meeting transcription, gated on speech and paid for by one participantJeremy Magland 18src/transcribe/
19 capture.ts AudioWorklet → 40 ms linear16 frames + RMS; SpeechGate (the VAD)
20 deepgram.ts one streaming WebSocket per speaker; subprotocol auth, KeepAlive
21 store.ts the transcript: paragraph merging, per-room localStorage persistence
22 transcriber.ts one capture+socket pipeline per participant, reconciled per snapshot
23 *.test.mjs node checks for the gate and the store (see Testing)
1ebfe6fAdd spotlight view, room chat, and clearer media-failure messagesJeremy Magland 24src/App.tsx landing form (light) + in-room view (dark), video grid with
25 click-to-spotlight (gallery ↔ one big tile + filmstrip; Esc or
8107f0eOptional meeting transcription, gated on speech and paid for by one participantJeremy Magland 26 click again to return), control bar, chat and transcript side
27 panels (one at a time; docked wide, overlay ≤700px)
06b5f04Optional TURN relay, gated by a token and shared across the roomJeremy Magland 28worker/ Cloudflare Worker that mints TURN credentials — deployed
29 separately (wrangler), NOT part of `npm run build`
32## Key design decisions
34- **Rooms, no registry.** The room ID is any string (whitespace stripped,
35 exact otherwise, case-sensitive); `roomTopic` hashes it into the nostr
36 presence topic. The URL hash holds the room (`#<encoded-room>`) so the
37 address bar is the invite link.
38- **Auto-mesh, no consent handshake.** Unlike commoncall, entering the room IS
39 the consent: on every presence announcement, `maybeConnect` brings up a
40 `Peer` (initiator = smaller peer ID, commonview's stalled-connection retry
41 at 15 s). All of commoncall's call-request/accept machinery is gone.
42- **Muted by default; placeholder tracks.** getUserMedia runs at entry but
43 tracks start `enabled = false`. Every participant ALWAYS carries exactly one
44 audio + one video track (denied/missing devices get a silent
45 AudioContext-destination track / black canvas-capture track), so
46 offer/answer stays symmetric and the one-offer, no-renegotiation design
47 holds. Unmuting without a real device retries getUserMedia and upgrades the
1ebfe6fAdd spotlight view, room chat, and clearer media-failure messagesJeremy Magland 48 placeholder via `replaceTrack` on every connection. getUserMedia failures
49 surface a cause-specific notice (`mediaErrorMessage`: permission vs
50 not-found vs device-busy, error name included) both at join and on retry —
51 on Linux, a camera held by another browser fails with NotReadableError,
52 which is NOT a permissions problem. The combined audio+video request fails
53 as a whole in that case, so `acquireMedia` retries each kind separately.
9813683Serverless group video calls: rooms, WebRTC mesh, shared settingsJeremy Magland 54- **Soft cap of 8** (`MAX_PARTICIPANTS`). A peer already holding 7 connections
55 answers an unknown peer's announcement/offer with `{t:'room-full'}` on the
56 newcomer's topic instead of connecting; a newcomer with zero connections
57 that receives room-full tears down and shows a notice. Two simultaneous
58 joiners racing for the last slot can briefly exceed the cap — accepted.
59- **Settings are room-wide, multi-party LWW.** One entry per key in
60 `settingsMeta` (`{rev, by}`); changes broadcast `{t:'set', key, value, rev,
61 by}` to all peers (complete graph — no relaying), late joiners get every
62 entry inside each peer's `hello`, and a same-rev tie is won by the SMALLER
63 setter ID. Default quality is `medium` — so quality caps are applied to each
64 sender on connect (`applyVideoParamsTo`, with one delayed retry because
65 encodings may not exist right at 'connected'), not only on change.
66- **Mute is per-participant, NOT a shared setting** — same as commoncall: own
67 flags, `{t:'mute'}` notices, `track.enabled` toggling, and the notice
68 carries the EFFECTIVE outgoing video state (screen share overrides camera
69 mute). Remote participants are assumed muted until told otherwise.
70- **Screen share = track swap on every connection.** `getDisplayMedia` +
71 `replaceTrack` per peer; a peer that joins mid-share gets the screen track
72 from `outgoingStream()`. Same-kind replacement avoids renegotiation — never
73 addTrack mid-connection.
1ebfe6fAdd spotlight view, room chat, and clearer media-failure messagesJeremy Magland 74- **Chat is ephemeral and never relayed.** `{t:'chat', text}` broadcasts on
75 the control channels; every message arrives directly from its author over a
76 channel established via signed signaling, so authorship needs no extra
77 crypto. There is deliberately NO history replay for late joiners — replay
78 would mean peers relaying others' messages, which a malicious peer could
79 fabricate; adding history requires signing each message. Log capped at 500,
80 messages at 2000 chars. Join/left lines are derived locally: hello carries a
81 self-reported `joinedAt`, and a peer whose join predates ours gets no
82 "joined" line on first sight (they were already here) — but `chatSeen`
83 ensures a blip-reconnect logs "joined" to match its "left". Links: only
84 http(s) URLs matched by `withLinks` become anchors (target=_blank,
85 rel=noopener noreferrer); never linkify other schemes.
06b5f04Optional TURN relay, gated by a token and shared across the roomJeremy Magland 86- **TURN is optional, token-gated, and shared room-wide.** `VITE_TURN_ENDPOINT`
87 (build time) points at `worker/`; unset ⇒ the whole feature is hidden and
88 `BASE_ICE_SERVERS` (STUN + openrelay) is used, as before. One participant
89 enters a token on the landing form, `mintIce` exchanges it for an
90 `{iceServers, expiresAt}` and it is broadcast as `{t:'ice'}` — on `connect`
91 to each peer, and again on refresh (`scheduleIceRefresh`, 5 min before
92 expiry). Peers `adoptIce` it; our own credentials always beat a shared one
93 (only we can refresh them), otherwise the later `expiresAt` wins. Share the
94 CREDENTIAL, never the token — the token never leaves the browser it was
95 typed into. **Peer-supplied ICE is untrusted:** `sanitizeIceConfig` bounds
96 and scheme-checks it before it reaches `RTCPeerConnection`.
97- **Credentials can't help the connection that carried them.** `Peer` takes
98 `iceServers` at construction and never renegotiates them, so a peer learns
99 credentials from the first peer it reaches and uses them for the NEXT
100 connection. Stalled pairs recover via the existing `CONNECT_RETRY_MS` path,
101 which calls `iceServers()` afresh — deliberately no proactive teardown on
102 adoption, since rebuilding a half-open pair out of step with the other side
103 is exactly what that retry already handles.
8107f0eOptional meeting transcription, gated on speech and paid for by one participantJeremy Magland 104- **Transcription: one participant pays, everyone is told.** Whoever enters a
105 Deepgram key transcribes the WHOLE room from their own browser — a mesh call
106 already delivers everyone's audio locally, so `syncTranscriptionSources`
107 (called from `rebuildSnapshot`) opens one pipeline per participant plus self.
108 Attribution is structural: one socket per speaker, so there is no
109 diarization to misread. The transcript is deliberately NOT broadcast — a
110 transcriber relaying text attributed to others is text they cannot vouch
111 for, the same objection that blocks chat history replay. What IS broadcast
112 is `{t:'tx', on}` (plus a `transcribing` flag in `hello` for late joiners),
113 driving a tile badge and chat lines. It is NOT a room setting: nobody else
114 can turn it on or off. The key lives in localStorage
115 (`commonroom:deepgramKey`) and, unlike the TURN token, has nothing shareable
116 derived from it.
117- **Streaming, not batch, because of CORS.** Deepgram's pre-recorded REST
118 endpoint is cheaper per minute but sends no CORS headers, so a browser needs
119 a proxy; the WebSocket endpoint connects directly. Browsers cannot set an
120 Authorization header on a WebSocket, so the key rides the subprotocol
121 (`new WebSocket(url, ['token', key])`) — Deepgram's documented client-side
122 scheme. A failed WS handshake hides its HTTP status, so a bad key and a dead
123 network are indistinguishable; `DeepgramStream` gives up after two
124 never-opened sockets rather than retrying on every utterance.
125- **Cost is the whole point of the VAD.** Billing follows audio sent, so
126 `SpeechGate` gates it and `KeepAlive` (every 3-5 s; the server times out at
127 10 s) holds the socket open through silence unbilled. Watch three things if
128 you retune it: the noise floor adapts ONLY while the gate is shut (else a
129 long utterance raises the bar against itself); it is seeded from the first
130 frame and held shut for 1 s (else someone joining from a noisy room opens it
131 on frame one and never closes); and a rise is capped at `4 × floor` per step
132 so one door slam cannot deafen it while a sustained change is still learned
133 in about a second. `MAX_OPEN_FRAMES` is the backstop for a gate stuck open.
134 A 320 ms pre-roll ring buffer means word onsets survive the gate opening.
135- **The transcript persists per room, the chat does not.** `TranscriptStore`
136 keys on the room ID (`commonroom:transcript:<room>`), restores on entry
137 before transcription even starts, debounces writes at 2 s, flushes on
138 `pagehide` and on leave, and trims from the front at 200k chars. Hence the
139 Clear button, which confirms inline — it is the one thing here that was
140 being kept on purpose.
9813683Serverless group video calls: rooms, WebRTC mesh, shared settingsJeremy Magland 141- **Cleanup is join-generation-guarded.** `joinSeq` is bumped on every
142 join/leave; async work (getUserMedia, topic hashing, display capture)
143 re-checks it after each await. `leave()` unsubscribes topics, stops all
144 tracks, closes the AudioContext, and resets settings to defaults.
146## Testing
148`npm run dev`, then open the room in two browsers (identity is
149per-browser-profile via localStorage, so two tabs in one profile are the SAME
150peer — use a private window or second browser). `npm run build` type-checks
151(`tsc -b`) and bundles. Let the user test multi-party media in real browsers;
152don't try to automate camera/mic flows.
154The Worker CAN be tested without a browser: `cd worker && cp .dev.vars.example
155.dev.vars && npm run dev`, then curl it. With the example values, token checks
156work (`goodtoken` passes, anything else 401s) and the upstream call 404s, which
157surfaces as a 502 — enough to cover auth, CORS and method handling. `cd worker
158&& npx tsc --noEmit` type-checks it; the root `tsc -b` does not (it only
159includes `src`). The sanitizers in `turn.ts` are pure and testable under node
160via `npx esbuild src/p2p/turn.ts --format=esm --define:import.meta.env='{}'`.
8107f0eOptional meeting transcription, gated on speech and paid for by one participantJeremy Magland 161
162The two riskiest pieces of the transcription path have node checks, by the
163same bundle-then-run recipe (there is no test runner in this repo; run them by
164hand after touching either file):
166```sh
167npx esbuild src/transcribe/capture.ts --format=esm --outfile=/tmp/capture.mjs
168node src/transcribe/gate.test.mjs /tmp/capture.mjs # VAD: what gets paid for
169npx esbuild src/transcribe/store.ts --format=esm --outfile=/tmp/store.mjs
170node src/transcribe/store.test.mjs /tmp/store.mjs # persistence: what must not be lost
171```
173What they cannot cover is whether Deepgram accepts the audio at all — that
174needs a real key and a real browser. The transcript panel's "N s sent" readout
175is the quickest check that gating works: it should climb while someone talks
176and sit still while nobody does.
06b5f04Optional TURN relay, gated by a token and shared across the roomJeremy Magland 177Whether a relay is actually USED can only be seen in a real browser
178(chrome://webrtc-internals, candidate pair type `relay`).