1# CLAUDE.md
3Tips for future agents working in this repo. It combines the p2p techniques of
4the sibling projects `commonview` (auto-connecting mesh) and `commoncall`
5(WebRTC media, settings, screen share) — read those first; this file only
6covers what is different here.
8## Architecture
10```
11src/p2p/
12 identity.ts schnorr keypair; pubkey hex = peer ID — ported from commoncall
13 nostr.ts minimal relay client + topic scheme — ported (roomTopic takes a room ID)
14 peer.ts WebRTC wrapper: media + control channel — ported (replaceTrack generalized to audio|video)
15 settings.ts shared ROOM settings, quality presets — default quality is 'medium', not 'auto'
16 turn.ts optional TURN: build-time endpoint, credential fetch, sanitizers
17 network.ts the heart: rooms, presence, mesh, media, settings sync, relay sharing
18src/transcribe/
19 capture.ts AudioWorklet → 40 ms linear16 frames + RMS; SpeechGate (the VAD)
20 deepgram.ts one streaming WebSocket per speaker; subprotocol auth, KeepAlive
21 store.ts the transcript: paragraph merging, per-room localStorage persistence
22 transcriber.ts one capture+socket pipeline per participant, reconciled per snapshot
23 *.test.mjs node checks for the gate and the store (see Testing)
24src/App.tsx landing form (light) + in-room view (dark), video grid with
25 click-to-spotlight (gallery ↔ one big tile + filmstrip; Esc or
26 click again to return), control bar, chat and transcript side
27 panels (one at a time; docked wide, overlay ≤700px)
28worker/ Cloudflare Worker that mints TURN credentials — deployed
29 separately (wrangler), NOT part of `npm run build`
30```
32## Key design decisions
34- **Rooms, no registry.** The room ID is any string (whitespace stripped,
35 exact otherwise, case-sensitive); `roomTopic` hashes it into the nostr
36 presence topic. The URL hash holds the room (`#<encoded-room>`) so the
37 address bar is the invite link.
38- **Auto-mesh, no consent handshake.** Unlike commoncall, entering the room IS
39 the consent: on every presence announcement, `maybeConnect` brings up a
40 `Peer` (initiator = smaller peer ID, commonview's stalled-connection retry
41 at 15 s). All of commoncall's call-request/accept machinery is gone.
42- **Muted by default; placeholder tracks.** getUserMedia runs at entry but
43 tracks start `enabled = false`. Every participant ALWAYS carries exactly one
44 audio + one video track (denied/missing devices get a silent
45 AudioContext-destination track / black canvas-capture track), so
46 offer/answer stays symmetric and the one-offer, no-renegotiation design
47 holds. Unmuting without a real device retries getUserMedia and upgrades the
48 placeholder via `replaceTrack` on every connection. getUserMedia failures
49 surface a cause-specific notice (`mediaErrorMessage`: permission vs
50 not-found vs device-busy, error name included) both at join and on retry —
51 on Linux, a camera held by another browser fails with NotReadableError,
52 which is NOT a permissions problem. The combined audio+video request fails
53 as a whole in that case, so `acquireMedia` retries each kind separately.
54- **Soft cap of 8** (`MAX_PARTICIPANTS`). A peer already holding 7 connections
55 answers an unknown peer's announcement/offer with `{t:'room-full'}` on the
56 newcomer's topic instead of connecting; a newcomer with zero connections
57 that receives room-full tears down and shows a notice. Two simultaneous
58 joiners racing for the last slot can briefly exceed the cap — accepted.
59- **Settings are room-wide, multi-party LWW.** One entry per key in
60 `settingsMeta` (`{rev, by}`); changes broadcast `{t:'set', key, value, rev,
61 by}` to all peers (complete graph — no relaying), late joiners get every
62 entry inside each peer's `hello`, and a same-rev tie is won by the SMALLER
63 setter ID. Default quality is `medium` — so quality caps are applied to each
64 sender on connect (`applyVideoParamsTo`, with one delayed retry because
65 encodings may not exist right at 'connected'), not only on change.
66- **Mute is per-participant, NOT a shared setting** — same as commoncall: own
67 flags, `{t:'mute'}` notices, `track.enabled` toggling, and the notice
68 carries the EFFECTIVE outgoing video state (screen share overrides camera
69 mute). Remote participants are assumed muted until told otherwise.
70- **Screen share = track swap on every connection.** `getDisplayMedia` +
71 `replaceTrack` per peer; a peer that joins mid-share gets the screen track
72 from `outgoingStream()`. Same-kind replacement avoids renegotiation — never
73 addTrack mid-connection.
74- **Chat is ephemeral and never relayed.** `{t:'chat', text}` broadcasts on
75 the control channels; every message arrives directly from its author over a
76 channel established via signed signaling, so authorship needs no extra
77 crypto. There is deliberately NO history replay for late joiners — replay
78 would mean peers relaying others' messages, which a malicious peer could
79 fabricate; adding history requires signing each message. Log capped at 500,
80 messages at 2000 chars. Join/left lines are derived locally: hello carries a
81 self-reported `joinedAt`, and a peer whose join predates ours gets no
82 "joined" line on first sight (they were already here) — but `chatSeen`
83 ensures a blip-reconnect logs "joined" to match its "left". Links: only
84 http(s) URLs matched by `withLinks` become anchors (target=_blank,
85 rel=noopener noreferrer); never linkify other schemes.
86- **TURN is optional, token-gated, and shared room-wide.** `VITE_TURN_ENDPOINT`
87 (build time) points at `worker/`; unset ⇒ the whole feature is hidden and
88 `BASE_ICE_SERVERS` (STUN + openrelay) is used, as before. One participant
89 enters a token on the landing form, `mintIce` exchanges it for an
90 `{iceServers, expiresAt}` and it is broadcast as `{t:'ice'}` — on `connect`
91 to each peer, and again on refresh (`scheduleIceRefresh`, 5 min before
92 expiry). Peers `adoptIce` it; our own credentials always beat a shared one
93 (only we can refresh them), otherwise the later `expiresAt` wins. Share the
94 CREDENTIAL, never the token — the token never leaves the browser it was
95 typed into. **Peer-supplied ICE is untrusted:** `sanitizeIceConfig` bounds
96 and scheme-checks it before it reaches `RTCPeerConnection`.
97- **Credentials can't help the connection that carried them.** `Peer` takes
98 `iceServers` at construction and never renegotiates them, so a peer learns
99 credentials from the first peer it reaches and uses them for the NEXT
100 connection. Stalled pairs recover via the existing `CONNECT_RETRY_MS` path,
101 which calls `iceServers()` afresh — deliberately no proactive teardown on
102 adoption, since rebuilding a half-open pair out of step with the other side
103 is exactly what that retry already handles.
104- **Transcription: one participant pays, everyone is told.** Whoever enters a
105 Deepgram key transcribes the WHOLE room from their own browser — a mesh call
106 already delivers everyone's audio locally, so `syncTranscriptionSources`
107 (called from `rebuildSnapshot`) opens one pipeline per participant plus self.
108 Attribution is structural: one socket per speaker, so there is no
109 diarization to misread. The transcript is deliberately NOT broadcast — a
110 transcriber relaying text attributed to others is text they cannot vouch
111 for, the same objection that blocks chat history replay. What IS broadcast
112 is `{t:'tx', on}` (plus a `transcribing` flag in `hello` for late joiners),
113 driving a tile badge and chat lines. It is NOT a room setting: nobody else
114 can turn it on or off. The key lives in localStorage
115 (`commonroom:deepgramKey`) and, unlike the TURN token, has nothing shareable
116 derived from it.
117- **Streaming, not batch, because of CORS.** Deepgram's pre-recorded REST
118 endpoint is cheaper per minute but sends no CORS headers, so a browser needs
119 a proxy; the WebSocket endpoint connects directly. Browsers cannot set an
120 Authorization header on a WebSocket, so the key rides the subprotocol
121 (`new WebSocket(url, ['token', key])`) — Deepgram's documented client-side
122 scheme. A failed WS handshake hides its HTTP status, so a bad key and a dead
123 network are indistinguishable; `DeepgramStream` gives up after two
124 never-opened sockets rather than retrying on every utterance.
125- **Cost is the whole point of the VAD.** Billing follows audio sent, so
126 `SpeechGate` gates it and `KeepAlive` (every 3-5 s; the server times out at
127 10 s) holds the socket open through silence unbilled. Watch three things if
128 you retune it: the noise floor adapts ONLY while the gate is shut (else a
129 long utterance raises the bar against itself); it is seeded from the first
130 frame and held shut for 1 s (else someone joining from a noisy room opens it
131 on frame one and never closes); and a rise is capped at `4 × floor` per step
132 so one door slam cannot deafen it while a sustained change is still learned
133 in about a second. `MAX_OPEN_FRAMES` is the backstop for a gate stuck open.
134 A 320 ms pre-roll ring buffer means word onsets survive the gate opening.
135- **The transcript persists per room, the chat does not.** `TranscriptStore`
136 keys on the room ID (`commonroom:transcript:<room>`), restores on entry
137 before transcription even starts, debounces writes at 2 s, flushes on
138 `pagehide` and on leave, and trims from the front at 200k chars. Hence the
139 Clear button, which confirms inline — it is the one thing here that was
140 being kept on purpose.
141- **Cleanup is join-generation-guarded.** `joinSeq` is bumped on every
142 join/leave; async work (getUserMedia, topic hashing, display capture)
143 re-checks it after each await. `leave()` unsubscribes topics, stops all
144 tracks, closes the AudioContext, and resets settings to defaults.
146## Testing
148`npm run dev`, then open the room in two browsers (identity is
149per-browser-profile via localStorage, so two tabs in one profile are the SAME
150peer — use a private window or second browser). `npm run build` type-checks
151(`tsc -b`) and bundles. Let the user test multi-party media in real browsers;
152don't try to automate camera/mic flows.
154The Worker CAN be tested without a browser: `cd worker && cp .dev.vars.example
155.dev.vars && npm run dev`, then curl it. With the example values, token checks
156work (`goodtoken` passes, anything else 401s) and the upstream call 404s, which
157surfaces as a 502 — enough to cover auth, CORS and method handling. `cd worker
158&& npx tsc --noEmit` type-checks it; the root `tsc -b` does not (it only
159includes `src`). The sanitizers in `turn.ts` are pure and testable under node
160via `npx esbuild src/p2p/turn.ts --format=esm --define:import.meta.env='{}'`.
162The two riskiest pieces of the transcription path have node checks, by the
163same bundle-then-run recipe (there is no test runner in this repo; run them by
164hand after touching either file):
166```sh
167npx esbuild src/transcribe/capture.ts --format=esm --outfile=/tmp/capture.mjs
168node src/transcribe/gate.test.mjs /tmp/capture.mjs # VAD: what gets paid for
169npx esbuild src/transcribe/store.ts --format=esm --outfile=/tmp/store.mjs
170node src/transcribe/store.test.mjs /tmp/store.mjs # persistence: what must not be lost
171```
173What they cannot cover is whether Deepgram accepts the audio at all — that
174needs a real key and a real browser. The transcript panel's "N s sent" readout
175is the quickest check that gating works: it should climb while someone talks
176and sit still while nobody does.
177Whether a relay is actually USED can only be seen in a real browser
178(chrome://webrtc-internals, candidate pair type `relay`).