Optional meeting transcription, gated on speech and paid for by one participant
Anyone in the room with a Deepgram API key can turn on a live transcript
from the new panel in the control bar. A mesh call already delivers every
participant's audio to every other one, so a single browser can transcribe
the whole room: it opens a separate streaming connection per speaker, which
makes attribution structural rather than inferred, with no diarization to
misread.
Deepgram's pre-recorded REST endpoint is cheaper per minute but sends no
CORS headers, so a browser cannot call it without a proxy; the streaming
WebSocket connects directly, which is what keeps the page serverless.
Browsers cannot set an Authorization header on a WebSocket either, so the
key rides the subprotocol as ['token', key], Deepgram's documented
client-side scheme. Note that a failed WebSocket handshake hides its HTTP
status, so a rejected key and an unreachable network are indistinguishable
from the client; the stream gives up after two never-opened sockets rather
than reconnecting into a wall on every utterance.
Billing follows the audio sent, so only speech is sent. A voice-activity
gate on each participant's audio streams while someone is talking and holds
the socket open through the silence with unbilled KeepAlive messages, which
is what keeps an hour-long call from being billed as eight hours of room
tone. A 320 ms pre-roll buffer means opening the gate does not clip the
first word, and the panel reports the seconds actually sent so the cost is
visible while it accrues. The gate's noise floor adapts only while it is
shut, is seeded from the first frame and held shut for a second at the
start, and rises by at most a factor of four per step; without the first a
long utterance would raise the bar against itself, without the second a
participant in a noisy room would open the gate on their first frame and
never close it, and without the third a slammed door would deafen it.
The transcript is deliberately not sent to the other participants. Text
attributed to someone but relayed by someone else is text they cannot vouch
for, which is the same objection that keeps chat history from being
replayed. What is shared is the fact that it is running, as a badge on the
transcriber's tile and a line in the chat, and it is not a room setting:
nobody else can turn it on or off. The key stays in the browser it was
typed into and has nothing shareable derived from it.
Unlike the chat, the transcript persists: it is stored under the room it
belongs to and restored on entry, with buttons to copy it, save it as text,
or clear it, the last of which confirms first. This does leave meeting
transcripts in local storage, which the README says plainly.
The gate and the store are the two pieces where a quiet mistake is
expensive, so both have node checks; see the Testing section of CLAUDE.md
for the bundle-then-run recipe.
11 changed files+1937−52
CLAUDE.mdmodified+61−2View file
@@ -15,10 +15,16 @@ src/p2p/
1515 settings.ts shared ROOM settings, quality presets — default quality is 'medium', not 'auto'
1616 turn.ts optional TURN: build-time endpoint, credential fetch, sanitizers
1717 network.ts the heart: rooms, presence, mesh, media, settings sync, relay sharing
18+src/transcribe/
19+ capture.ts AudioWorklet → 40 ms linear16 frames + RMS; SpeechGate (the VAD)
20+ deepgram.ts one streaming WebSocket per speaker; subprotocol auth, KeepAlive
21+ store.ts the transcript: paragraph merging, per-room localStorage persistence
22+ transcriber.ts one capture+socket pipeline per participant, reconciled per snapshot
23+ *.test.mjs node checks for the gate and the store (see Testing)
1824 src/App.tsx landing form (light) + in-room view (dark), video grid with
1925 click-to-spotlight (gallery ↔ one big tile + filmstrip; Esc or
20- click again to return), control bar, chat panel (side panel on
21- wide screens, overlay ≤700px, unread badge)
26+ click again to return), control bar, chat and transcript side
27+ panels (one at a time; docked wide, overlay ≤700px)
2228 worker/ Cloudflare Worker that mints TURN credentials — deployed
2329 separately (wrangler), NOT part of `npm run build`
2430 ```
@@ -95,6 +101,43 @@ worker/ Cloudflare Worker that mints TURN credentials — deployed
95101 which calls `iceServers()` afresh — deliberately no proactive teardown on
96102 adoption, since rebuilding a half-open pair out of step with the other side
97103 is exactly what that retry already handles.
104+- **Transcription: one participant pays, everyone is told.** Whoever enters a
105+ Deepgram key transcribes the WHOLE room from their own browser — a mesh call
106+ already delivers everyone's audio locally, so `syncTranscriptionSources`
107+ (called from `rebuildSnapshot`) opens one pipeline per participant plus self.
108+ Attribution is structural: one socket per speaker, so there is no
109+ diarization to misread. The transcript is deliberately NOT broadcast — a
110+ transcriber relaying text attributed to others is text they cannot vouch
111+ for, the same objection that blocks chat history replay. What IS broadcast
112+ is `{t:'tx', on}` (plus a `transcribing` flag in `hello` for late joiners),
113+ driving a tile badge and chat lines. It is NOT a room setting: nobody else
114+ can turn it on or off. The key lives in localStorage
115+ (`commonroom:deepgramKey`) and, unlike the TURN token, has nothing shareable
116+ derived from it.
117+- **Streaming, not batch, because of CORS.** Deepgram's pre-recorded REST
118+ endpoint is cheaper per minute but sends no CORS headers, so a browser needs
119+ a proxy; the WebSocket endpoint connects directly. Browsers cannot set an
120+ Authorization header on a WebSocket, so the key rides the subprotocol
121+ (`new WebSocket(url, ['token', key])`) — Deepgram's documented client-side
122+ scheme. A failed WS handshake hides its HTTP status, so a bad key and a dead
123+ network are indistinguishable; `DeepgramStream` gives up after two
124+ never-opened sockets rather than retrying on every utterance.
125+- **Cost is the whole point of the VAD.** Billing follows audio sent, so
126+ `SpeechGate` gates it and `KeepAlive` (every 3-5 s; the server times out at
127+ 10 s) holds the socket open through silence unbilled. Watch three things if
128+ you retune it: the noise floor adapts ONLY while the gate is shut (else a
129+ long utterance raises the bar against itself); it is seeded from the first
130+ frame and held shut for 1 s (else someone joining from a noisy room opens it
131+ on frame one and never closes); and a rise is capped at `4 × floor` per step
132+ so one door slam cannot deafen it while a sustained change is still learned
133+ in about a second. `MAX_OPEN_FRAMES` is the backstop for a gate stuck open.
134+ A 320 ms pre-roll ring buffer means word onsets survive the gate opening.
135+- **The transcript persists per room, the chat does not.** `TranscriptStore`
136+ keys on the room ID (`commonroom:transcript:<room>`), restores on entry
137+ before transcription even starts, debounces writes at 2 s, flushes on
138+ `pagehide` and on leave, and trims from the front at 200k chars. Hence the
139+ Clear button, which confirms inline — it is the one thing here that was
140+ being kept on purpose.
98141 - **Cleanup is join-generation-guarded.** `joinSeq` is bumped on every
99142 join/leave; async work (getUserMedia, topic hashing, display capture)
100143 re-checks it after each await. `leave()` unsubscribes topics, stops all
@@ -115,5 +158,21 @@ surfaces as a 502 — enough to cover auth, CORS and method handling. `cd worker
115158 && npx tsc --noEmit` type-checks it; the root `tsc -b` does not (it only
116159 includes `src`). The sanitizers in `turn.ts` are pure and testable under node
117160 via `npx esbuild src/p2p/turn.ts --format=esm --define:import.meta.env='{}'`.
161+
162+The two riskiest pieces of the transcription path have node checks, by the
163+same bundle-then-run recipe (there is no test runner in this repo; run them by
164+hand after touching either file):
165+
166+```sh
167+npx esbuild src/transcribe/capture.ts --format=esm --outfile=/tmp/capture.mjs
168+node src/transcribe/gate.test.mjs /tmp/capture.mjs # VAD: what gets paid for
169+npx esbuild src/transcribe/store.ts --format=esm --outfile=/tmp/store.mjs
170+node src/transcribe/store.test.mjs /tmp/store.mjs # persistence: what must not be lost
171+```
172+
173+What they cannot cover is whether Deepgram accepts the audio at all — that
174+needs a real key and a real browser. The transcript panel's "N s sent" readout
175+is the quickest check that gating works: it should climb while someone talks
176+and sit still while nobody does.
118177 Whether a relay is actually USED can only be seen in a real browser
119178 (chrome://webrtc-internals, candidate pair type `relay`).
README.mdmodified+32−1View file
@@ -16,7 +16,8 @@ everyone — currently the video quality (low / medium / high / auto, medium by
1616 default). You can also share your screen in place of your camera, and there's
1717 a room chat (with join/left notices and clickable links) that is as ephemeral
1818 as the call itself: you only see what's said while you're in the room, and
19-nothing is stored anywhere.
19+nothing is stored anywhere. Anyone with a Deepgram API key can also transcribe
20+the meeting from their own browser (see below).
2021
2122 ## How it works
2223
@@ -49,6 +50,36 @@ quality presets, screen share):
4950 map to `RTCRtpSender.setParameters` caps that each participant applies to
5051 its own outgoing senders.
5152
53+## Transcribing a meeting
54+
55+Anyone in the room who has a [Deepgram](https://deepgram.com) API key can turn
56+on a live transcript from the transcript panel in the control bar. Because a
57+mesh call already delivers everyone's audio to every participant, that one
58+browser can transcribe the whole room: it opens a separate streaming
59+connection per speaker, so the transcript says who said what without relying on
60+speaker diarization to guess.
61+
62+Only speech is sent. A voice-activity gate on each participant's audio holds
63+the connection open through silence with unbilled keep-alive messages and
64+streams audio only while someone is actually talking, which is what keeps an
65+hour-long call from being billed as eight hours of room tone. The panel shows
66+how many seconds have actually been sent, so the cost is visible while it
67+accrues. A short pre-roll buffer means the gate opening does not clip the
68+first word.
69+
70+The key is stored in this browser and used only to connect to Deepgram; it is
71+never shared with the other participants, and the person who enters it is the
72+one billed. The transcript is likewise local: it is **not** sent to the other
73+participants, for the same reason the chat has no history replay — text
74+attributed to someone but relayed by someone else is text they cannot vouch
75+for. Everyone does see that transcription is running, as a badge on the
76+transcriber's tile and a line in the chat.
77+
78+Unlike the chat, the transcript is kept: it is stored in this browser under the
79+room's name and comes back the next time you enter that room, with buttons to
80+copy it, save it as a text file, or clear it (which asks first). Bear in mind
81+that this leaves meeting transcripts in the browser's local storage.
82+
5283 ## The relay, and how one token covers a room
5384
5485 Most pairs of browsers can reach each other directly once STUN has told them
src/App.tsxmodified+450−46View file
@@ -6,6 +6,7 @@ import {
66 } from './p2p/network'
77 import {VIDEO_QUALITIES, type VideoQuality} from './p2p/settings'
88 import {TURN_CONFIGURED, type RelayStatus} from './p2p/turn'
9+import {formatTranscript, type TranscriptItem} from './transcribe/store'
910 import {useNetwork} from './useNetwork'
1011
1112 // Screen capture is desktop-only in practice; hide the button where the API
@@ -177,6 +178,28 @@ const ICONS = {
177178 <path d="M22 2L11 13" />
178179 <path d="M22 2l-7 20-4-9-9-4 20-7z" />
179180 </>
181+ ),
182+ transcript: (
183+ <>
184+ <path d="M14 2H6a2 2 0 0 0-2 2v16a2 2 0 0 0 2 2h12a2 2 0 0 0 2-2V8z" />
185+ <path d="M14 2v6h6" />
186+ <path d="M16 13H8" />
187+ <path d="M16 17H8" />
188+ <path d="M10 9H8" />
189+ </>
190+ ),
191+ copy: (
192+ <>
193+ <rect x="9" y="9" width="13" height="13" rx="2" />
194+ <path d="M5 15H4a2 2 0 0 1-2-2V4a2 2 0 0 1 2-2h9a2 2 0 0 1 2 2v1" />
195+ </>
196+ ),
197+ download: (
198+ <>
199+ <path d="M21 15v4a2 2 0 0 1-2 2H5a2 2 0 0 1-2-2v-4" />
200+ <path d="M7 10l5 5 5-5" />
201+ <path d="M12 15V3" />
202+ </>
180203 )
181204 } as const
182205
@@ -252,6 +275,7 @@ function Tile({
252275 audioMuted,
253276 videoMuted,
254277 connecting,
278+ transcribing,
255279 fill = false,
256280 onClick,
257281 tooltip
@@ -263,6 +287,8 @@ function Tile({
263287 audioMuted: boolean
264288 videoMuted: boolean
265289 connecting: boolean
290+ /** They are sending the room's audio to a transcription service. */
291+ transcribing: boolean
266292 /** Fill the parent box (spotlight) instead of a fixed 4:3 grid cell. */
267293 fill?: boolean
268294 onClick?: () => void
@@ -326,6 +352,29 @@ function Tile({
326352 )}
327353 </div>
328354 )}
355+ {transcribing && (
356+ <span
357+ title={`${isSelf ? 'You are' : `${label} is`} transcribing this meeting — everyone's audio is being sent to a transcription service.`}
358+ style={{
359+ ...tileChip,
360+ position: 'absolute',
361+ left: 8,
362+ top: 8,
363+ background: 'rgba(160, 20, 20, 0.85)'
364+ }}
365+ >
366+ <span
367+ style={{
368+ width: 7,
369+ height: 7,
370+ borderRadius: '50%',
371+ background: '#fff',
372+ flex: '0 0 auto'
373+ }}
374+ />
375+ transcribing
376+ </span>
377+ )}
329378 <div
330379 style={{
331380 position: 'absolute',
@@ -597,6 +646,49 @@ function withLinks(text: string): React.ReactNode[] {
597646 const timeLabel = (t: number): string =>
598647 new Date(t).toLocaleTimeString([], {hour: '2-digit', minute: '2-digit'})
599648
649+// Side panels (chat, transcript) share their geometry: docked beside the
650+// gallery on wide screens, floating over it on narrow ones. Only one is open
651+// at a time, so they never compete for the space.
652+const panelStyle = (overlay: boolean): React.CSSProperties => ({
653+ ...(overlay
654+ ? {
655+ position: 'absolute',
656+ top: 0,
657+ right: 0,
658+ bottom: 0,
659+ width: 'min(320px, 88vw)',
660+ zIndex: 10,
661+ boxShadow: '-4px 0 16px rgba(0, 0, 0, 0.5)'
662+ }
663+ : {width: 300, flex: '0 0 auto'}),
664+ background: '#161616',
665+ borderLeft: '1px solid #333',
666+ display: 'flex',
667+ flexDirection: 'column',
668+ minHeight: 0
669+})
670+
671+const panelHeadStyle: React.CSSProperties = {
672+ display: 'flex',
673+ justifyContent: 'space-between',
674+ alignItems: 'center',
675+ padding: '0.45rem 0.45rem 0.45rem 0.75rem',
676+ borderBottom: '1px solid #333'
677+}
678+
679+function PanelClose({onClose, what}: {onClose: () => void; what: string}) {
680+ return (
681+ <button
682+ style={{...iconBtn, padding: '0.35rem', border: 'none', background: 'none'}}
683+ onClick={onClose}
684+ title={`Close ${what}`}
685+ aria-label={`Close ${what}`}
686+ >
687+ <Icon name="x" size={16} />
688+ </button>
689+ )
690+}
691+
600692 function ChatPanel({
601693 items,
602694 overlay,
@@ -626,44 +718,10 @@ function ChatPanel({
626718 }
627719
628720 return (
629- <aside
630- style={{
631- ...(overlay
632- ? {
633- position: 'absolute',
634- top: 0,
635- right: 0,
636- bottom: 0,
637- width: 'min(320px, 88vw)',
638- zIndex: 10,
639- boxShadow: '-4px 0 16px rgba(0, 0, 0, 0.5)'
640- }
641- : {width: 300, flex: '0 0 auto'}),
642- background: '#161616',
643- borderLeft: '1px solid #333',
644- display: 'flex',
645- flexDirection: 'column',
646- minHeight: 0
647- }}
648- >
649- <div
650- style={{
651- display: 'flex',
652- justifyContent: 'space-between',
653- alignItems: 'center',
654- padding: '0.45rem 0.45rem 0.45rem 0.75rem',
655- borderBottom: '1px solid #333'
656- }}
657- >
721+ <aside style={panelStyle(overlay)}>
722+ <div style={panelHeadStyle}>
658723 <strong>Chat</strong>
659- <button
660- style={{...iconBtn, padding: '0.35rem', border: 'none', background: 'none'}}
661- onClick={onClose}
662- title="Close chat"
663- aria-label="Close chat"
664- >
665- <Icon name="x" size={16} />
666- </button>
724+ <PanelClose onClose={onClose} what="chat" />
667725 </div>
668726 <div
669727 ref={listRef}
@@ -755,6 +813,307 @@ function ChatPanel({
755813 )
756814 }
757815
816+// ---- transcript -------------------------------------------------------------
817+
818+const smallBtn: React.CSSProperties = {
819+ ...darkBtn,
820+ padding: '0.35rem 0.6rem',
821+ fontSize: '0.85rem'
822+}
823+
824+const durationLabel = (seconds: number): string =>
825+ seconds < 60
826+ ? `${Math.round(seconds)} s`
827+ : `${(seconds / 60).toFixed(1)} min`
828+
829+function TranscriptPanel({
830+ items,
831+ seconds,
832+ active,
833+ hasKey,
834+ overlay,
835+ onStart,
836+ onStop,
837+ onClear,
838+ onForgetKey,
839+ onClose
840+}: {
841+ items: TranscriptItem[]
842+ seconds: number
843+ active: boolean
844+ hasKey: boolean
845+ overlay: boolean
846+ onStart: (apiKey: string) => void
847+ onStop: () => void
848+ onClear: () => void
849+ onForgetKey: () => void
850+ onClose: () => void
851+}) {
852+ const [key, setKey] = useState('')
853+ const [copied, setCopied] = useState(false)
854+ // Clearing throws away something that was deliberately kept, so it asks
855+ // first — inline, rather than in a dialog that would cover the transcript
856+ // being discussed.
857+ const [confirmClear, setConfirmClear] = useState(false)
858+ useEffect(() => {
859+ if (items.length === 0) setConfirmClear(false)
860+ }, [items.length])
861+ const listRef = useRef<HTMLDivElement>(null)
862+ const stickRef = useRef(true)
863+ useEffect(() => {
864+ const el = listRef.current
865+ if (el && stickRef.current) el.scrollTop = el.scrollHeight
866+ }, [items.length])
867+
868+ const start = (e: React.FormEvent) => {
869+ e.preventDefault()
870+ onStart(key)
871+ setKey('')
872+ }
873+
874+ const copy = async () => {
875+ try {
876+ await navigator.clipboard.writeText(formatTranscript(items))
877+ setCopied(true)
878+ setTimeout(() => setCopied(false), 1500)
879+ } catch {
880+ /* clipboard unavailable; the text is still selectable in the panel */
881+ }
882+ }
883+
884+ const save = () => {
885+ const url = URL.createObjectURL(
886+ new Blob([formatTranscript(items)], {type: 'text/plain'})
887+ )
888+ const a = document.createElement('a')
889+ a.href = url
890+ a.download = `transcript-${new Date()
891+ .toISOString()
892+ .slice(0, 16)
893+ .replace(/[:T]/g, '-')}.txt`
894+ a.click()
895+ setTimeout(() => URL.revokeObjectURL(url), 1000)
896+ }
897+
898+ return (
899+ <aside style={panelStyle(overlay)}>
900+ <div style={panelHeadStyle}>
901+ <strong>Transcript</strong>
902+ <PanelClose onClose={onClose} what="the transcript" />
903+ </div>
904+
905+ <div
906+ style={{
907+ padding: '0.6rem 0.75rem',
908+ borderBottom: '1px solid #333',
909+ fontSize: '0.85rem',
910+ color: '#bbb'
911+ }}
912+ >
913+ {active ? (
914+ <div
915+ style={{display: 'flex', alignItems: 'center', gap: '0.5rem'}}
916+ >
917+ <span
918+ style={{
919+ width: 9,
920+ height: 9,
921+ borderRadius: '50%',
922+ background: '#e05252',
923+ flex: '0 0 auto'
924+ }}
925+ />
926+ <span style={{flex: 1}}>
927+ Listening · {durationLabel(seconds)} sent
928+ </span>
929+ <button style={smallBtn} onClick={onStop}>
930+ Stop
931+ </button>
932+ </div>
933+ ) : hasKey ? (
934+ <div style={{display: 'flex', alignItems: 'center', gap: '0.5rem'}}>
935+ <span style={{flex: 1}}>
936+ Everyone in the room will be told that you are transcribing.
937+ </span>
938+ <button style={smallBtn} onClick={() => onStart('')}>
939+ Start
940+ </button>
941+ </div>
942+ ) : (
943+ <form onSubmit={start}>
944+ <p style={{margin: '0 0 0.5rem'}}>
945+ Transcription runs from this browser through Deepgram, so it
946+ needs your own API key. Only speech is sent, not the silence in
947+ between, and each person is transcribed separately so the
948+ transcript says who said what.
949+ </p>
950+ <input
951+ type="password"
952+ value={key}
953+ onChange={e => setKey(e.target.value)}
954+ placeholder="Deepgram API key"
955+ autoComplete="off"
956+ style={{
957+ width: '100%',
958+ boxSizing: 'border-box',
959+ padding: '0.45rem 0.6rem',
960+ borderRadius: 8,
961+ border: '1px solid #555',
962+ background: '#2a2a2a',
963+ color: '#eee',
964+ fontSize: '0.9rem'
965+ }}
966+ />
967+ <button
968+ type="submit"
969+ style={
970+ key.trim()
971+ ? {...smallBtn, marginTop: '0.5rem'}
972+ : {...smallBtn, marginTop: '0.5rem', ...disabledStyle}
973+ }
974+ disabled={!key.trim()}
975+ >
976+ Start transcribing
977+ </button>
978+ <p style={{margin: '0.5rem 0 0', color: '#888'}}>
979+ The key is stored in this browser and is never sent to the other
980+ participants. You are the one billed for it.
981+ </p>
982+ </form>
983+ )}
984+ </div>
985+
986+ <div
987+ ref={listRef}
988+ onScroll={() => {
989+ const el = listRef.current
990+ if (el) {
991+ stickRef.current =
992+ el.scrollHeight - el.scrollTop - el.clientHeight < 60
993+ }
994+ }}
995+ style={{
996+ flex: 1,
997+ minHeight: 0,
998+ overflowY: 'auto',
999+ padding: '0.6rem 0.75rem'
1000+ }}
1001+ >
1002+ {items.length === 0 && (
1003+ <p style={{color: '#777', fontSize: '0.85rem'}}>
1004+ {active
1005+ ? 'Nothing transcribed yet. Speech will appear here a moment after someone says it.'
1006+ : "No transcript yet. What gets transcribed is kept in this browser under this room's name and will be here next time; nobody else receives it."}
1007+ </p>
1008+ )}
1009+ {items.map(item => (
1010+ <div key={item.seq} style={{marginBottom: '0.7rem'}}>
1011+ <div style={{fontSize: '0.75rem', color: '#888'}}>
1012+ <strong style={{color: '#ccc'}}>{item.name}</strong>{' '}
1013+ {timeLabel(item.time)}
1014+ </div>
1015+ <div
1016+ style={{
1017+ wordBreak: 'break-word',
1018+ fontSize: '0.92rem',
1019+ lineHeight: 1.45
1020+ }}
1021+ >
1022+ {item.text}
1023+ </div>
1024+ </div>
1025+ ))}
1026+ </div>
1027+
1028+ {confirmClear ? (
1029+ <div
1030+ style={{
1031+ display: 'flex',
1032+ gap: '0.4rem',
1033+ alignItems: 'center',
1034+ padding: '0.6rem',
1035+ borderTop: '1px solid #333',
1036+ background: '#2a1a1a'
1037+ }}
1038+ >
1039+ <span style={{flex: 1, fontSize: '0.85rem'}}>
1040+ Delete this room's transcript?
1041+ </span>
1042+ <button
1043+ style={{
1044+ ...smallBtn,
1045+ background: '#c62828',
1046+ borderColor: '#c62828',
1047+ color: '#fff'
1048+ }}
1049+ onClick={() => {
1050+ onClear()
1051+ setConfirmClear(false)
1052+ }}
1053+ >
1054+ Delete
1055+ </button>
1056+ <button style={smallBtn} onClick={() => setConfirmClear(false)}>
1057+ Cancel
1058+ </button>
1059+ </div>
1060+ ) : (
1061+ <div
1062+ style={{
1063+ display: 'flex',
1064+ gap: '0.4rem',
1065+ alignItems: 'center',
1066+ padding: '0.6rem',
1067+ borderTop: '1px solid #333'
1068+ }}
1069+ >
1070+ <button
1071+ style={items.length ? smallBtn : {...smallBtn, ...disabledStyle}}
1072+ disabled={items.length === 0}
1073+ onClick={() => void copy()}
1074+ title="Copy the transcript"
1075+ >
1076+ <Icon name="copy" size={14} />
1077+ {copied ? 'Copied!' : 'Copy'}
1078+ </button>
1079+ <button
1080+ style={items.length ? smallBtn : {...smallBtn, ...disabledStyle}}
1081+ disabled={items.length === 0}
1082+ onClick={save}
1083+ title="Save the transcript as a text file"
1084+ >
1085+ <Icon name="download" size={14} />
1086+ Save
1087+ </button>
1088+ <button
1089+ style={items.length ? smallBtn : {...smallBtn, ...disabledStyle}}
1090+ disabled={items.length === 0}
1091+ onClick={() => setConfirmClear(true)}
1092+ title="Delete this room's stored transcript"
1093+ >
1094+ Clear
1095+ </button>
1096+ <span style={{flex: 1}} />
1097+ {hasKey && !active && (
1098+ <button
1099+ style={{
1100+ ...smallBtn,
1101+ border: 'none',
1102+ background: 'none',
1103+ color: '#999'
1104+ }}
1105+ onClick={onForgetKey}
1106+ title="Remove the stored Deepgram key from this browser"
1107+ >
1108+ Forget key
1109+ </button>
1110+ )}
1111+ </div>
1112+ )}
1113+ </aside>
1114+ )
1115+}
1116+
7581117 export default function App() {
7591118 const {snapshot, network} = useNetwork()
7601119 const {
@@ -771,6 +1130,10 @@ export default function App() {
7711130 settings,
7721131 chat,
7731132 relay,
1133+ transcribing,
1134+ transcript,
1135+ transcriptSeconds,
1136+ hasDeepgramKey,
7741137 notice
7751138 } = snapshot
7761139
@@ -795,15 +1158,16 @@ export default function App() {
7951158 return () => window.removeEventListener('keydown', onKey)
7961159 }, [focus])
7971160
798- // Chat panel: side panel on wide screens, overlay on narrow ones. The
799- // unread badge counts real messages (not join/left lines) that arrived
800- // while the panel was closed.
801- const [chatOpen, setChatOpen] = useState(false)
1161+ // One side panel at a time: docked on wide screens, an overlay on narrow
1162+ // ones. The chat's unread badge counts real messages (not join/left lines)
1163+ // that arrived while the chat was not the open panel.
1164+ const [panel, setPanel] = useState<'chat' | 'transcript' | null>(null)
1165+ const chatOpen = panel === 'chat'
8021166 const [readCount, setReadCount] = useState(0)
8031167 const narrow = useMediaQuery('(max-width: 700px)')
8041168 useEffect(() => {
805- if (phase !== 'room' && chatOpen) setChatOpen(false)
806- }, [phase, chatOpen])
1169+ if (phase !== 'room' && panel !== null) setPanel(null)
1170+ }, [phase, panel])
8071171 useEffect(() => {
8081172 // Follows the log while open; also snaps back when the log resets on leave.
8091173 if ((chatOpen || readCount > chat.length) && readCount !== chat.length) {
@@ -864,6 +1228,7 @@ export default function App() {
8641228 audioMuted={audioMuted}
8651229 videoMuted={videoMuted && !sharing}
8661230 connecting={false}
1231+ transcribing={transcribing}
8671232 fill={focused}
8681233 tooltip={tileTooltip(focused)}
8691234 onClick={() => setFocusedId(focused ? null : 'self')}
@@ -878,6 +1243,7 @@ export default function App() {
8781243 audioMuted={p.audioMuted}
8791244 videoMuted={p.videoMuted}
8801245 connecting={!p.connected}
1246+ transcribing={p.transcribing}
8811247 fill={focused}
8821248 tooltip={tileTooltip(focused)}
8831249 onClick={() => setFocusedId(focused ? null : p.peerId)}
@@ -1075,12 +1441,26 @@ export default function App() {
10751441 </>
10761442 )}
10771443 </main>
1078- {chatOpen && (
1444+ {panel === 'chat' && (
10791445 <ChatPanel
10801446 items={chat}
10811447 overlay={narrow}
10821448 onSend={text => network.sendChat(text)}
1083- onClose={() => setChatOpen(false)}
1449+ onClose={() => setPanel(null)}
1450+ />
1451+ )}
1452+ {panel === 'transcript' && (
1453+ <TranscriptPanel
1454+ items={transcript}
1455+ seconds={transcriptSeconds}
1456+ active={transcribing}
1457+ hasKey={hasDeepgramKey}
1458+ overlay={narrow}
1459+ onStart={key => void network.startTranscription(key)}
1460+ onStop={() => network.stopTranscription()}
1461+ onClear={() => network.clearTranscript()}
1462+ onForgetKey={() => network.forgetDeepgramKey()}
1463+ onClose={() => setPanel(null)}
10841464 />
10851465 )}
10861466 </div>
@@ -1144,7 +1524,7 @@ export default function App() {
11441524 style={chatOpen ? {...iconBtn, background: '#3a3a3a'} : iconBtn}
11451525 title={chatOpen ? 'Close the chat' : 'Open the chat'}
11461526 aria-label={chatOpen ? 'Close the chat' : 'Open the chat'}
1147- onClick={() => setChatOpen(!chatOpen)}
1527+ onClick={() => setPanel(chatOpen ? null : 'chat')}
11481528 >
11491529 <Icon name="chat" />
11501530 </button>
@@ -1171,6 +1551,30 @@ export default function App() {
11711551 </span>
11721552 )}
11731553 </span>
1554+ <button
1555+ style={
1556+ transcribing
1557+ ? {...iconBtn, background: '#a01414', borderColor: '#a01414', color: '#fff'}
1558+ : panel === 'transcript'
1559+ ? {...iconBtn, background: '#3a3a3a'}
1560+ : iconBtn
1561+ }
1562+ title={
1563+ transcribing
1564+ ? 'Transcribing — open the transcript'
1565+ : 'Transcribe this meeting'
1566+ }
1567+ aria-label={
1568+ transcribing
1569+ ? 'Transcribing; open the transcript'
1570+ : 'Transcribe this meeting'
1571+ }
1572+ onClick={() =>
1573+ setPanel(panel === 'transcript' ? null : 'transcript')
1574+ }
1575+ >
1576+ <Icon name="transcript" />
1577+ </button>
11741578 <label
11751579 title="Video quality for the whole room — anyone can change it, and it applies to everyone"
11761580 style={{
src/p2p/network.tsmodified+184−2View file
@@ -17,6 +17,9 @@ import {
1717 type IceConfig,
1818 type RelayStatus
1919 } from './turn'
20+import {isUsableApiKey} from '../transcribe/deepgram'
21+import {TranscriptStore, type TranscriptItem} from '../transcribe/store'
22+import {Transcriber, type TranscriptSource} from '../transcribe/transcriber'
2023
2124 // ---------------------------------------------------------------------------
2225 // CommonRoom network layer: a full-mesh group video call.
@@ -67,11 +70,14 @@ type ControlMsg =
6770 * can tell "was already here when I arrived" from "joined after me"
6871 * for the chat's join lines. */
6972 joinedAt: number
73+ transcribing: boolean
7074 settings: SettingEntry[]
7175 }
7276 | ({t: 'set'} & SettingEntry)
7377 | {t: 'mute'; audio: boolean; video: boolean}
7478 | {t: 'chat'; text: string}
79+ /** Whether we are sending the room's audio to a transcription service. */
80+ | {t: 'tx'; on: boolean}
7581 /** TURN credentials, so one person's token covers the whole room. */
7682 | {t: 'ice'; iceServers: RTCIceServer[]; expiresAt: number}
7783 | {t: 'bye'}
@@ -86,6 +92,7 @@ const CONNECT_RETRY_MS = 15000
8692
8793 const NAME_KEY = 'commonroom:name'
8894 const TURN_TOKEN_KEY = 'commonroom:turnToken'
95+const DEEPGRAM_KEY = 'commonroom:deepgramKey'
8996
9097 /** Re-mint our relay credentials this long before they lapse, so a long call
9198 * never runs out mid-session. */
@@ -108,6 +115,8 @@ interface Conn {
108115 * — everyone starts muted). */
109116 audioMuted: boolean
110117 videoMuted: boolean
118+ /** Whether they told us they are transcribing the room. */
119+ transcribing: boolean
111120 }
112121
113122 export interface ChatItem {
@@ -129,6 +138,7 @@ export interface ParticipantInfo {
129138 stream: MediaStream | null
130139 audioMuted: boolean
131140 videoMuted: boolean
141+ transcribing: boolean
132142 }
133143
134144 export interface Snapshot {
@@ -148,6 +158,17 @@ export interface Snapshot {
148158 chat: ChatItem[]
149159 /** Where our TURN credentials came from (ours, a peer's, or none). */
150160 relay: RelayStatus
161+ /** Whether WE are transcribing (peers report their own in participants). */
162+ transcribing: boolean
163+ /** This room's transcript, restored from previous sittings and appended to
164+ * while transcription runs. */
165+ transcript: TranscriptItem[]
166+ /** Audio sent to Deepgram for this room, in seconds — what it is billed on.
167+ * Cumulative across sittings, like the transcript itself. */
168+ transcriptSeconds: number
169+ /** Whether a Deepgram key is stored in this browser. The key itself never
170+ * reaches the UI, and never leaves this browser. */
171+ hasDeepgramKey: boolean
151172 notice: string | null
152173 }
153174
@@ -200,6 +221,16 @@ export class Network {
200221 * network blip gets a "joined" line to match its "left" line). */
201222 private chatSeen = new Set<string>()
202223
224+ // Transcription is entirely local: it belongs to whoever entered a Deepgram
225+ // key, it is not a room setting, and the only thing that crosses the mesh is
226+ // the fact that it is running. The transcript itself outlives the call and
227+ // belongs to the room, so its store is opened on entry whether or not
228+ // anything is being transcribed this time.
229+ private transcript: TranscriptStore | null = null
230+ private transcriber: Transcriber | null = null
231+ private transcribing = false
232+ private deepgramKey = localStorage.getItem(DEEPGRAM_KEY) ?? ''
233+
203234 private notice: string | null = null
204235
205236 private snapshot!: Snapshot
@@ -220,6 +251,9 @@ export class Network {
220251 // presence TTL when a tab closes.
221252 window.addEventListener('pagehide', () => {
222253 if (this.phase === 'room') this.broadcastControl({t: 'bye'})
254+ // Closing the tab is the most likely way to leave, and the transcript
255+ // is the one thing here meant to survive it.
256+ this.transcript?.flush()
223257 })
224258 }
225259
@@ -242,6 +276,9 @@ export class Network {
242276 /* ignore */
243277 }
244278 this.notice = null
279+ // Whatever was transcribed in this room before is part of the room, so it
280+ // is back on screen from the moment you enter.
281+ this.transcript = new TranscriptStore(rm, () => this.rebuildSnapshot())
245282 this.phase = 'joining'
246283 this.rebuildSnapshot()
247284 const seq = ++this.joinSeq
@@ -355,6 +392,14 @@ export class Network {
355392 // shared, belong to whoever was in that room — don't carry them onward.
356393 this.ice = null
357394 this.iceFromSelf = false
395+ this.transcribing = false
396+ if (this.transcriber) {
397+ this.transcriber.dispose()
398+ this.transcriber = null
399+ }
400+ // The transcript stays on disk under its room; only the open handle goes.
401+ this.transcript?.flush()
402+ this.transcript = null
358403 if (this.screenStream) {
359404 for (const t of this.screenStream.getTracks()) t.stop()
360405 this.screenStream = null
@@ -506,7 +551,8 @@ export class Network {
506551 connected: false,
507552 stream: null,
508553 audioMuted: true,
509- videoMuted: true
554+ videoMuted: true,
555+ transcribing: false
510556 }
511557 this.conns.set(peerId, conn)
512558
@@ -695,6 +741,7 @@ export class Network {
695741 audioMuted: this.audioMuted,
696742 videoMuted: this.effectiveVideoMuted(),
697743 joinedAt: this.joinedAtMs,
744+ transcribing: this.transcribing,
698745 settings
699746 } satisfies ControlMsg)
700747 )
@@ -713,6 +760,7 @@ export class Network {
713760 if (typeof msg.name === 'string') conn.name = msg.name.slice(0, 40)
714761 conn.audioMuted = msg.audioMuted !== false
715762 conn.videoMuted = msg.videoMuted !== false
763+ conn.transcribing = msg.transcribing === true
716764 if (Array.isArray(msg.settings)) {
717765 for (const entry of msg.settings) this.applyRemoteSetting(entry)
718766 }
@@ -730,6 +778,11 @@ export class Network {
730778 this.chatPresent.set(peerId, name)
731779 this.chatSeen.add(peerId)
732780 if (!preexisting) this.pushSystem(`${name} joined`)
781+ // Walking into a room that is already being transcribed is exactly
782+ // the case where nobody has seen the announcement, so say it here.
783+ if (conn.transcribing) {
784+ this.pushSystem(`${name} is transcribing this meeting`)
785+ }
733786 }
734787 this.rebuildSnapshot()
735788 return
@@ -761,6 +814,19 @@ export class Network {
761814 this.rebuildSnapshot()
762815 return
763816 }
817+ case 'tx': {
818+ if (typeof msg.on !== 'boolean' || conn.transcribing === msg.on) return
819+ conn.transcribing = msg.on
820+ const name =
821+ this.presence.get(peerId)?.name ?? conn.name ?? peerId.slice(0, 8)
822+ this.pushSystem(
823+ msg.on
824+ ? `${name} started transcribing this meeting`
825+ : `${name} stopped transcribing`
826+ )
827+ this.rebuildSnapshot()
828+ return
829+ }
764830 case 'ice': {
765831 // Untrusted input: a peer could send anything here, so the list is
766832 // validated down to well-formed ICE URLs before it goes near a
@@ -1045,6 +1111,116 @@ export class Network {
10451111 }
10461112 }
10471113
1114+ // ---- transcription ---------------------------------------------------------
1115+ //
1116+ // Whoever has a Deepgram key can transcribe the room from their own browser,
1117+ // since a mesh call already delivers everyone's audio to everyone. The key
1118+ // stays in that browser: unlike the relay token, there is nothing to share,
1119+ // because the transcription is done by one participant on behalf of all.
1120+ //
1121+ // Two things are deliberately NOT done here. The transcript is not sent to
1122+ // the other participants — a transcriber relaying text attributed to other
1123+ // people is text those people cannot vouch for, which is the same objection
1124+ // that keeps chat history from being replayed (see the chat section). And it
1125+ // is not a room setting: nobody else can turn it on or off. What IS shared is
1126+ // the fact that it is running, both as a badge on the tile and as a line in
1127+ // the chat, because recording people without telling them is not acceptable.
1128+
1129+ async startTranscription(apiKey: string) {
1130+ if (this.phase !== 'room' || this.transcribing) return
1131+ const key = apiKey.trim() || this.deepgramKey
1132+ if (!key) return
1133+ if (!isUsableApiKey(key)) {
1134+ this.notice =
1135+ 'That Deepgram API key contains characters that cannot be sent in a browser connection — check for spaces or line breaks.'
1136+ this.rebuildSnapshot()
1137+ return
1138+ }
1139+ if (key !== this.deepgramKey) {
1140+ this.deepgramKey = key
1141+ localStorage.setItem(DEEPGRAM_KEY, key)
1142+ }
1143+ const store = this.transcript
1144+ if (!store) return
1145+ if (!this.transcriber || this.transcriber.apiKey !== key) {
1146+ this.transcriber?.dispose()
1147+ this.transcriber = new Transcriber(
1148+ key,
1149+ store,
1150+ () => this.rebuildSnapshot(),
1151+ (message, fatal) => this.transcriptionFailed(message, fatal)
1152+ )
1153+ }
1154+ const tr = this.transcriber
1155+ const seq = this.joinSeq
1156+ const ok = await tr.start()
1157+ if (this.joinSeq !== seq || this.transcriber !== tr) {
1158+ tr.dispose()
1159+ return
1160+ }
1161+ if (!ok) {
1162+ this.rebuildSnapshot()
1163+ return
1164+ }
1165+ this.transcribing = true
1166+ this.broadcastControl({t: 'tx', on: true})
1167+ this.pushSystem('You started transcribing this meeting')
1168+ this.rebuildSnapshot()
1169+ }
1170+
1171+ stopTranscription() {
1172+ if (!this.transcribing) return
1173+ this.transcribing = false
1174+ this.transcriber?.stop()
1175+ if (this.phase === 'room') {
1176+ this.broadcastControl({t: 'tx', on: false})
1177+ this.pushSystem('You stopped transcribing')
1178+ }
1179+ this.rebuildSnapshot()
1180+ }
1181+
1182+ /** Drop the stored key. The transcript already produced is kept. */
1183+ forgetDeepgramKey() {
1184+ this.stopTranscription()
1185+ localStorage.removeItem(DEEPGRAM_KEY)
1186+ this.deepgramKey = ''
1187+ this.rebuildSnapshot()
1188+ }
1189+
1190+ /** Discard this room's transcript, here and on disk. The panel confirms
1191+ * first: unlike the chat, this is the one thing here that was being kept. */
1192+ clearTranscript() {
1193+ this.transcript?.clear()
1194+ this.rebuildSnapshot()
1195+ }
1196+
1197+ private transcriptionFailed(message: string, fatal: boolean) {
1198+ this.notice = message
1199+ // Unconditionally rebuild: stopTranscription is a no-op if we had already
1200+ // stopped, and the notice still has to reach the screen.
1201+ if (fatal) this.stopTranscription()
1202+ this.rebuildSnapshot()
1203+ }
1204+
1205+ /** Keep the transcriber's per-speaker pipelines in step with the room. This
1206+ * runs on every snapshot; the transcriber ignores sources it already has. */
1207+ private syncTranscriptionSources() {
1208+ const tr = this.transcriber
1209+ if (!tr?.active) return
1210+ const sources: TranscriptSource[] = [
1211+ {id: selfId, name: this.name ?? 'You', stream: this.localStream}
1212+ ]
1213+ for (const [peerId, conn] of this.conns) {
1214+ if (!conn.stream) continue
1215+ sources.push({
1216+ id: peerId,
1217+ name: this.presence.get(peerId)?.name ?? conn.name ?? peerId.slice(0, 8),
1218+ stream: conn.stream
1219+ })
1220+ }
1221+ tr.setSources(sources)
1222+ }
1223+
10481224 // ---- public API -------------------------------------------------------
10491225
10501226 /** Change the room-wide video-quality preset. Anyone can change it; every
@@ -1066,6 +1242,7 @@ export class Network {
10661242 }
10671243
10681244 private rebuildSnapshot() {
1245+ this.syncTranscriptionSources()
10691246 const ids = new Set<string>([...this.conns.keys(), ...this.presence.keys()])
10701247 const participants: ParticipantInfo[] = [...ids]
10711248 .map(peerId => {
@@ -1079,7 +1256,8 @@ export class Network {
10791256 connected: conn?.connected ?? false,
10801257 stream: conn?.stream ?? null,
10811258 audioMuted: conn?.audioMuted ?? true,
1082- videoMuted: conn?.videoMuted ?? true
1259+ videoMuted: conn?.videoMuted ?? true,
1260+ transcribing: conn?.transcribing ?? false
10831261 }
10841262 })
10851263 .sort(
@@ -1102,6 +1280,10 @@ export class Network {
11021280 settings: this.settings,
11031281 chat: this.chat,
11041282 relay: this.relayStatus(),
1283+ transcribing: this.transcribing,
1284+ transcript: this.transcript?.items ?? [],
1285+ transcriptSeconds: this.transcript?.audioSeconds ?? 0,
1286+ hasDeepgramKey: this.deepgramKey.length > 0,
11051287 notice: this.notice
11061288 }
11071289 for (const l of this.listeners) l()
src/transcribe/capture.tsadded+287−0View file
@@ -0,0 +1,287 @@
1+// Speech capture: turn one participant's audio into fixed-size linear16 frames
2+// and decide which of those frames actually carry speech.
3+//
4+// The point of the gate is cost. Deepgram bills the audio you send it, so an
5+// eight-person room left open for an hour would be billed for eight hours of
6+// mostly silence. Sending only while someone is talking cuts that to roughly
7+// the time people actually speak; the socket stays open across the gaps on
8+// KeepAlive messages instead (see deepgram.ts).
9+//
10+// The detector is deliberately simple: frame energy against an adaptive noise
11+// floor, with hysteresis in both level and time. We emphasize that this is not
12+// a speech/non-speech classifier — a slammed door or a burst of typing will
13+// open the gate, and whatever Deepgram makes of it (usually nothing) lands in
14+// the transcript. What it does do reliably is stay shut through ordinary room
15+// tone, which is where the savings are. A learned detector such as Silero
16+// would reject non-speech far better, at the cost of a few megabytes of model
17+// and runtime; that trade did not seem worth it here.
18+
19+/** Frame length. Short enough for responsive gating, long enough that the
20+ * message rate stays modest with eight participants. */
21+const FRAME_MS = 40
22+/** Audio retained before the gate opens, so word onsets are not clipped. */
23+const PREROLL_FRAMES = 8 // 320 ms
24+/** Consecutive loud frames before speech is declared. */
25+const OPEN_FRAMES = 2 // 80 ms
26+/** Consecutive quiet frames before it ends. Generous, because a mid-sentence
27+ * pause that closed the gate would clip the word after it. */
28+const CLOSE_FRAMES = 20 // 800 ms
29+// The noise floor adapts only while the gate is shut, so a long utterance can
30+// never raise the bar against itself. It falls quickly and rises slowly, since
31+// a level that persists is the room and a level that does not is a person.
32+const FALL_ALPHA = 0.3
33+const RISE_ALPHA = 0.05
34+/** A rise is capped at this multiple of the current estimate per step, so one
35+ * slammed door cannot lift the floor and deafen the gate behind it. Only a
36+ * sustained change in the room moves it. */
37+const RISE_CLAMP = 4
38+const OPEN_FACTOR = 2.5
39+const CLOSE_FACTOR = 1.6
40+/** Absolute floor on the open threshold (RMS of normalized samples). Without
41+ * it, a silent room whose noise estimate has decayed toward zero would open
42+ * on anything at all. */
43+const MIN_RMS = 0.005
44+/** Frames at the start, and after the watchdog below, during which the gate
45+ * is held shut so the floor can be learned before it is used. Without this
46+ * the estimate would start at zero, and someone joining from a noisy room
47+ * would open the gate on their first frame and never close it. */
48+const CALIBRATE_FRAMES = 25 // 1 s
49+/** A gate that has been open this long is almost certainly stuck on a room
50+ * that got loud after we learned it, which is the expensive failure. Force it
51+ * shut and re-learn. A genuine monologue with no 800 ms pause in a full
52+ * minute loses the second of audio that recalibration costs. */
53+const MAX_OPEN_FRAMES = 1500 // 60 s
54+
55+const WORKLET_NAME = 'commonroom-frames'
56+
57+// The worklet is shipped as a source string rather than a separate module so
58+// there is no extra build configuration: it becomes a blob URL at runtime.
59+// It does the framing and the energy measurement (both cheap, both per
60+// sample); the state machine lives on the main thread where it is easier to
61+// reason about and adjust.
62+const WORKLET_SRC = `
63+class FrameProcessor extends AudioWorkletProcessor {
64+ constructor(options) {
65+ super()
66+ this.size = options.processorOptions.frameSize
67+ this.buf = new Float32Array(this.size)
68+ this.n = 0
69+ this.done = false
70+ // Any message means "you are finished": returning false releases the
71+ // processor instead of leaving it in the graph for the rest of the call.
72+ this.port.onmessage = () => { this.done = true }
73+ }
74+ process(inputs) {
75+ if (this.done) return false
76+ const ch = inputs[0] && inputs[0][0]
77+ if (!ch) return true
78+ for (let i = 0; i < ch.length; i++) {
79+ this.buf[this.n++] = ch[i]
80+ if (this.n < this.size) continue
81+ const pcm = new Int16Array(this.size)
82+ let sum = 0
83+ for (let j = 0; j < this.size; j++) {
84+ let s = this.buf[j]
85+ if (s > 1) s = 1
86+ else if (s < -1) s = -1
87+ sum += s * s
88+ pcm[j] = s < 0 ? s * 0x8000 : s * 0x7fff
89+ }
90+ this.port.postMessage(
91+ {pcm: pcm, rms: Math.sqrt(sum / this.size)},
92+ [pcm.buffer]
93+ )
94+ this.n = 0
95+ }
96+ return true
97+ }
98+}
99+registerProcessor(${JSON.stringify(WORKLET_NAME)}, FrameProcessor)
100+`
101+
102+let workletUrl: string | null = null
103+/** addModule is per-context and must not be repeated, so the promise is cached
104+ * against the context rather than re-issued for every participant. */
105+const registered = new WeakMap<BaseAudioContext, Promise<void>>()
106+
107+export function registerCaptureWorklet(ctx: BaseAudioContext): Promise<void> {
108+ let p = registered.get(ctx)
109+ if (!p) {
110+ if (!workletUrl) {
111+ workletUrl = URL.createObjectURL(
112+ new Blob([WORKLET_SRC], {type: 'application/javascript'})
113+ )
114+ }
115+ p = ctx.audioWorklet.addModule(workletUrl)
116+ registered.set(ctx, p)
117+ }
118+ return p
119+}
120+
121+/** The gate on its own, with no audio plumbing: feed it frame energies and it
122+ * reports the frame speech starts on and the frame it ends on. Keeping it
123+ * separable is what makes the tuning above testable outside a browser. */
124+export class SpeechGate {
125+ private noise = 0
126+ private loud = 0
127+ private quiet = 0
128+ private open = 0
129+ private active = false
130+ private started = false
131+ private calibrating = CALIBRATE_FRAMES
132+
133+ get speaking(): boolean {
134+ return this.active
135+ }
136+
137+ /** The noise floor the thresholds are currently derived from. */
138+ get noiseFloor(): number {
139+ return this.noise
140+ }
141+
142+ push(rms: number): 'open' | 'close' | null {
143+ if (!this.started) {
144+ // Start from the room as we find it rather than from zero, so a loud
145+ // room is recognized as loud on the first frame instead of being
146+ // mistaken for an hour of speech.
147+ this.started = true
148+ this.noise = rms
149+ }
150+ if (this.calibrating > 0) {
151+ this.calibrating--
152+ this.track(rms)
153+ return null
154+ }
155+ if (!this.active) {
156+ this.track(rms)
157+ this.loud = rms >= Math.max(this.noise * OPEN_FACTOR, MIN_RMS)
158+ ? this.loud + 1
159+ : 0
160+ if (this.loud < OPEN_FRAMES) return null
161+ this.active = true
162+ this.quiet = 0
163+ this.open = 0
164+ return 'open'
165+ }
166+ this.open++
167+ this.quiet = rms < Math.max(this.noise * CLOSE_FACTOR, MIN_RMS * 0.7)
168+ ? this.quiet + 1
169+ : 0
170+ if (this.quiet >= CLOSE_FRAMES) {
171+ this.active = false
172+ this.loud = 0
173+ return 'close'
174+ }
175+ if (this.open >= MAX_OPEN_FRAMES) {
176+ this.active = false
177+ this.loud = 0
178+ this.calibrating = CALIBRATE_FRAMES
179+ return 'close'
180+ }
181+ return null
182+ }
183+
184+ /** Move the noise floor toward the current frame. Only called while the gate
185+ * is shut, so what it learns is the room and not the speaker. */
186+ private track(rms: number) {
187+ if (rms < this.noise) {
188+ this.noise = this.noise * (1 - FALL_ALPHA) + rms * FALL_ALPHA
189+ return
190+ }
191+ const target = Math.min(rms, this.noise * RISE_CLAMP + MIN_RMS)
192+ this.noise = this.noise * (1 - RISE_ALPHA) + target * RISE_ALPHA
193+ }
194+}
195+
196+export interface SpeechHandlers {
197+ /** A frame to transcribe, in order. Pre-roll frames arrive in a burst at the
198+ * moment speech is declared. */
199+ frame: (pcm: Int16Array) => void
200+ /** The gate shut — flush whatever the transcriber has buffered. */
201+ end: () => void
202+}
203+
204+/** Gated capture of one MediaStream's first audio track. Call
205+ * `registerCaptureWorklet(ctx)` and await it before constructing. */
206+export class SpeechCapture {
207+ private source: MediaStreamAudioSourceNode
208+ private node: AudioWorkletNode
209+ private sink: GainNode
210+ private gate = new SpeechGate()
211+ private ring: Int16Array[] = []
212+ private closed = false
213+
214+ constructor(
215+ ctx: AudioContext,
216+ stream: MediaStream,
217+ private handlers: SpeechHandlers
218+ ) {
219+ const frameSize = Math.round((ctx.sampleRate * FRAME_MS) / 1000)
220+ this.source = ctx.createMediaStreamSource(stream)
221+ this.node = new AudioWorkletNode(ctx, WORKLET_NAME, {
222+ numberOfInputs: 1,
223+ numberOfOutputs: 1,
224+ outputChannelCount: [1],
225+ processorOptions: {frameSize}
226+ })
227+ this.node.port.onmessage = e => {
228+ const {pcm, rms} = e.data as {pcm: Int16Array; rms: number}
229+ this.onFrame(pcm, rms)
230+ }
231+ // A worklet is only pulled if it reaches the destination, so it goes there
232+ // through a muted gain node — the participant's audio is already being
233+ // played by their <video> element and must not be played twice.
234+ this.sink = ctx.createGain()
235+ this.sink.gain.value = 0
236+ this.source.connect(this.node)
237+ this.node.connect(this.sink)
238+ this.sink.connect(ctx.destination)
239+ }
240+
241+ private onFrame(pcm: Int16Array, rms: number) {
242+ if (this.closed) return
243+ const event = this.gate.push(rms)
244+ if (event === 'open') {
245+ // The frame that tripped the gate, and the pre-roll behind it, are the
246+ // beginning of the utterance.
247+ this.ring.push(pcm)
248+ for (const f of this.ring) this.handlers.frame(f)
249+ this.ring = []
250+ return
251+ }
252+ if (event === 'close') {
253+ this.handlers.frame(pcm)
254+ this.handlers.end()
255+ return
256+ }
257+ if (this.gate.speaking) {
258+ this.handlers.frame(pcm)
259+ return
260+ }
261+ this.ring.push(pcm)
262+ if (this.ring.length > PREROLL_FRAMES) this.ring.shift()
263+ }
264+
265+ close() {
266+ if (this.closed) return
267+ this.closed = true
268+ this.node.port.onmessage = null
269+ // Ask the processor to retire itself. This is best effort — a disconnected
270+ // node is no longer pulled, so the stop may never be acted on — but a node
271+ // with no references and no connections is collectable either way.
272+ try {
273+ this.node.port.postMessage('stop')
274+ } catch {
275+ /* ignore */
276+ }
277+ // Disconnecting a node that was never fully connected throws in some
278+ // browsers; nothing here is worth failing a teardown over.
279+ try {
280+ this.source.disconnect()
281+ this.node.disconnect()
282+ this.sink.disconnect()
283+ } catch {
284+ /* ignore */
285+ }
286+ }
287+}
src/transcribe/deepgram.tsadded+235−0View file
@@ -0,0 +1,235 @@
1+// One Deepgram streaming connection, carrying one participant's speech.
2+//
3+// Deepgram offers three speech-to-text paths. The pre-recorded REST endpoint is
4+// the cheaper one per minute, but its responses carry no CORS headers, so a
5+// browser cannot call it without a proxy server; the streaming WebSocket
6+// endpoint is reachable directly. Since CommonRoom has no server of its own,
7+// streaming is what we use.
8+//
9+// Browsers cannot set an Authorization header on a WebSocket, so the API key
10+// travels as a subprotocol — `Sec-WebSocket-Protocol: token, <key>`, which is
11+// Deepgram's documented scheme for client-side connections. The key therefore
12+// leaves the browser only in the TLS handshake with Deepgram itself, and it is
13+// never shared with the room.
14+//
15+// The connection is opened lazily on the first frame of speech and then held
16+// open across pauses with KeepAlive messages, which are not billed. A separate
17+// connection per speaker is what gives the transcript its attribution: there is
18+// no diarization to interpret, since each socket only ever hears one person.
19+
20+const ENDPOINT = 'wss://api.deepgram.com/v1/listen'
21+
22+/** Deepgram closes an idle socket after 10 s; the docs ask for a KeepAlive
23+ * every 3-5 s. */
24+const TICK_MS = 3000
25+const KEEPALIVE_AFTER_MS = 2500
26+/** Give the socket back after a long silence rather than pinging it forever. */
27+const IDLE_CLOSE_MS = 120_000
28+/** Frames buffered while the socket is still opening (~4 s of speech). */
29+const QUEUE_CAP = 100
30+/** How long to wait for trailing results after asking the stream to close. */
31+const DRAIN_MS = 3000
32+
33+const QUERY = {
34+ model: 'nova-3',
35+ language: 'en',
36+ smart_format: 'true',
37+ // Our own gate already segments speech, and interim results would triple the
38+ // message rate for text we would only overwrite.
39+ interim_results: 'false',
40+ encoding: 'linear16',
41+ channels: '1',
42+ endpointing: '400'
43+}
44+
45+export interface DeepgramHandlers {
46+ transcript: (text: string) => void
47+ /** Samples actually written to the socket, for the cost readout. */
48+ sent: (samples: number) => void
49+ /** A problem worth surfacing. `fatal` means stop trying entirely. */
50+ failure: (message: string, fatal: boolean) => void
51+}
52+
53+/** A subprotocol must be an RFC 7230 token, which excludes spaces and most
54+ * punctuation. A key with a stray character would make the WebSocket
55+ * constructor throw rather than fail as a rejected key, so it is checked up
56+ * front where a comprehensible message can still be given. Deepgram keys are
57+ * hexadecimal and pass comfortably. */
58+export const isUsableApiKey = (key: string): boolean =>
59+ /^[A-Za-z0-9!#$%&'*+\-.^_`|~]+$/.test(key)
60+
61+export class DeepgramStream {
62+ private ws: WebSocket | null = null
63+ private queue: Int16Array[] = []
64+ private opened = false
65+ private failures = 0
66+ private lastAudioAt = 0
67+ private timer: number | null = null
68+ /** Set once the stream is winding down: no new audio, but results still
69+ * arriving from the server are delivered. */
70+ private draining = false
71+
72+ constructor(
73+ private apiKey: string,
74+ private sampleRate: number,
75+ private handlers: DeepgramHandlers
76+ ) {}
77+
78+ send(pcm: Int16Array) {
79+ if (this.draining) return
80+ this.lastAudioAt = Date.now()
81+ if (!this.ws) this.open()
82+ const ws = this.ws
83+ if (!ws) return
84+ if (ws.readyState === WebSocket.CONNECTING) {
85+ // Everything up to here is speech we decided was worth paying for, so it
86+ // waits for the socket rather than being dropped — but only up to a
87+ // point, in case the connection never comes up.
88+ if (this.queue.length < QUEUE_CAP) this.queue.push(pcm)
89+ return
90+ }
91+ if (ws.readyState !== WebSocket.OPEN) return
92+ ws.send(pcm)
93+ this.handlers.sent(pcm.length)
94+ }
95+
96+ /** End of utterance: ask Deepgram to flush what it has rather than waiting
97+ * for its own endpointing to expire. */
98+ finalize() {
99+ const ws = this.ws
100+ if (ws && ws.readyState === WebSocket.OPEN) {
101+ ws.send(JSON.stringify({type: 'Finalize'}))
102+ }
103+ }
104+
105+ /** Stop sending, but stay open briefly so the last utterance still lands. */
106+ stop() {
107+ if (this.draining) return
108+ this.draining = true
109+ this.queue = []
110+ if (this.timer !== null) clearInterval(this.timer)
111+ this.timer = null
112+ const ws = this.ws
113+ if (!ws) return
114+ if (ws.readyState === WebSocket.OPEN) {
115+ ws.send(JSON.stringify({type: 'CloseStream'}))
116+ window.setTimeout(() => ws.close(), DRAIN_MS)
117+ } else {
118+ ws.close()
119+ }
120+ }
121+
122+ private open() {
123+ const params = new URLSearchParams({
124+ ...QUERY,
125+ sample_rate: String(this.sampleRate)
126+ })
127+ let ws: WebSocket
128+ try {
129+ ws = new WebSocket(`${ENDPOINT}?${params}`, ['token', this.apiKey])
130+ } catch {
131+ this.handlers.failure(
132+ 'That Deepgram API key could not be used to open a connection — check it for stray characters.',
133+ true
134+ )
135+ return
136+ }
137+ ws.binaryType = 'arraybuffer'
138+ this.ws = ws
139+ this.opened = false
140+
141+ ws.onopen = () => {
142+ this.opened = true
143+ this.failures = 0
144+ for (const pcm of this.queue.splice(0)) {
145+ ws.send(pcm)
146+ this.handlers.sent(pcm.length)
147+ }
148+ }
149+ ws.onmessage = ev => this.onMessage(ev)
150+ // An error is always followed by a close, which carries more information.
151+ ws.onerror = () => undefined
152+ ws.onclose = ev => this.onClose(ws, ev)
153+
154+ if (this.timer === null) {
155+ this.timer = window.setInterval(() => this.tick(), TICK_MS)
156+ }
157+ }
158+
159+ private onMessage(ev: MessageEvent) {
160+ if (typeof ev.data !== 'string') return
161+ let msg: {
162+ type?: string
163+ channel?: {alternatives?: {transcript?: string}[]}
164+ description?: string
165+ message?: string
166+ }
167+ try {
168+ msg = JSON.parse(ev.data)
169+ } catch {
170+ return
171+ }
172+ if (msg.type === 'Results') {
173+ const text = msg.channel?.alternatives?.[0]?.transcript
174+ if (typeof text === 'string' && text.trim()) this.handlers.transcript(text)
175+ return
176+ }
177+ if (msg.type === 'Error') {
178+ const why = msg.description ?? msg.message ?? 'unspecified'
179+ this.handlers.failure(`Deepgram reported an error: ${why}`, false)
180+ }
181+ }
182+
183+ private onClose(ws: WebSocket, ev: CloseEvent) {
184+ // A socket we already gave up on (see the idle path in `tick`) may close
185+ // after its replacement is up; it has nothing left to say.
186+ if (this.ws !== ws) return
187+ const wasOpen = this.opened
188+ this.ws = null
189+ this.opened = false
190+ this.queue = []
191+ if (this.timer !== null) clearInterval(this.timer)
192+ this.timer = null
193+ if (this.draining) return
194+
195+ if (!wasOpen) {
196+ // A browser deliberately hides the HTTP status of a failed WebSocket
197+ // handshake, so a rejected key and an unreachable network are
198+ // indistinguishable here. Say so, and stop after the second attempt
199+ // rather than reconnecting into a wall on every utterance.
200+ this.failures++
201+ this.handlers.failure(
202+ `Could not open a Deepgram connection${
203+ ev.reason ? ` (${ev.reason})` : ''
204+ } — check the API key and your network.`,
205+ this.failures >= 2
206+ )
207+ return
208+ }
209+ if (ev.code !== 1000) {
210+ this.handlers.failure(
211+ `The Deepgram connection dropped${
212+ ev.reason ? ` (${ev.reason})` : ''
213+ }; it will reopen when someone next speaks.`,
214+ false
215+ )
216+ }
217+ }
218+
219+ private tick() {
220+ const ws = this.ws
221+ if (!ws || ws.readyState !== WebSocket.OPEN) return
222+ const idle = Date.now() - this.lastAudioAt
223+ if (idle > IDLE_CLOSE_MS) {
224+ // Nothing can be in flight after two minutes of silence, so this one is
225+ // released outright; the next utterance opens a fresh socket.
226+ ws.send(JSON.stringify({type: 'CloseStream'}))
227+ ws.close()
228+ this.ws = null
229+ if (this.timer !== null) clearInterval(this.timer)
230+ this.timer = null
231+ return
232+ }
233+ if (idle > KEEPALIVE_AFTER_MS) ws.send(JSON.stringify({type: 'KeepAlive'}))
234+ }
235+}
src/transcribe/gate.test.mjsadded+147−0View file
@@ -0,0 +1,147 @@
1+// Checks on the voice-activity gate in capture.ts. The gate decides what gets
2+// paid for, and its constants are the kind that invite tweaking, so its
3+// behavior is pinned here. It is pure arithmetic over frame energies, so it
4+// runs under node once capture.ts is bundled:
5+//
6+// npx esbuild src/transcribe/capture.ts --format=esm --outfile=/tmp/capture.mjs
7+// node src/transcribe/gate.test.mjs /tmp/capture.mjs
8+//
9+// Energies are RMS of samples normalized to [-1, 1]; frames are 40 ms.
10+
11+const {SpeechGate} = await import(process.argv[2] ?? '/tmp/capture.mjs')
12+
13+let failed = 0
14+const check = (label, ok, detail = '') => {
15+ if (!ok) failed++
16+ console.log(`${ok ? 'ok ' : 'FAIL'} ${label}${detail ? ` — ${detail}` : ''}`)
17+}
18+
19+// Feed a sequence of frame energies and record where the gate opened/closed.
20+const run = frames => {
21+ const g = new SpeechGate()
22+ const events = []
23+ frames.forEach((rms, i) => {
24+ const e = g.push(rms)
25+ if (e) events.push([e, i])
26+ })
27+ return {events, gate: g}
28+}
29+
30+const rep = (v, n) => Array(n).fill(v)
31+
32+// 1. Digital silence never opens the gate (a muted participant, or the silent
33+// placeholder track someone without a microphone sends).
34+check('silence stays shut', run(rep(0, 500)).events.length === 0)
35+
36+// 2. Ordinary quiet room tone never opens it either.
37+const roomTone = Array.from({length: 500}, () => 0.0015 + Math.random() * 0.001)
38+check('room tone stays shut', run(roomTone).events.length === 0)
39+
40+// 3. Speech opens it, and within OPEN_FRAMES (2) of onset.
41+const speech = [...rep(0.002, 100), ...rep(0.05, 50), ...rep(0.002, 100)]
42+{
43+ const {events} = run(speech)
44+ const open = events.find(e => e[0] === 'open')
45+ const close = events.find(e => e[0] === 'close')
46+ check('speech opens the gate', !!open, open && `at frame ${open[1]}`)
47+ check('opens promptly', open && open[1] - 100 <= 2, open && `${open[1] - 100} frames late`)
48+ check('speech closes the gate', !!close, close && `at frame ${close[1]}`)
49+ // CLOSE_FRAMES is 20 (800 ms) after the last loud frame at index 149.
50+ check('closes after the hold', close && close[1] - 149 === 20, close && `${close[1] - 149} frames`)
51+}
52+
53+// 4. A mid-sentence pause shorter than the hold does NOT split the utterance.
54+{
55+ const {events} = run([
56+ ...rep(0.002, 60),
57+ ...rep(0.05, 25),
58+ ...rep(0.002, 15), // 600 ms pause, under the 800 ms hold
59+ ...rep(0.05, 25),
60+ ...rep(0.002, 60)
61+ ])
62+ check(
63+ 'a short pause does not split the utterance',
64+ events.filter(e => e[0] === 'open').length === 1,
65+ `${events.filter(e => e[0] === 'open').length} openings`
66+ )
67+}
68+
69+// 5. A noisy room raises the floor: the same absolute level that counts as
70+// speech in a quiet room is ignored once it IS the room.
71+{
72+ const noisy = rep(0.02, 400)
73+ const {events, gate} = run(noisy)
74+ check('steady noise is learned, not transcribed', events.length === 0)
75+ check('noise floor tracked up', gate.noiseFloor > 0.015, `floor ${gate.noiseFloor.toFixed(4)}`)
76+ // Speech must now clear 2.5x the floor to register.
77+ const g2 = new SpeechGate()
78+ rep(0.02, 400).forEach(v => g2.push(v))
79+ const over = [0.08, 0.08, 0.08].map(v => g2.push(v))
80+ check('speech above a noisy floor still opens', over.includes('open'))
81+}
82+
83+// 6. The floor does not creep up during a long utterance and shut it off.
84+{
85+ const {events} = run([...rep(0.002, 50), ...rep(0.06, 600), ...rep(0.002, 50)])
86+ check(
87+ 'a long utterance is not cut short',
88+ events.filter(e => e[0] === 'open').length === 1 &&
89+ events.filter(e => e[0] === 'close').length === 1
90+ )
91+}
92+
93+// 7. One isolated loud frame (a click) does not open it.
94+{
95+ const {events} = run([...rep(0.002, 50), 0.2, ...rep(0.002, 50)])
96+ check('a single click does not open the gate', events.length === 0)
97+}
98+
99+// 8. The room gets loud AFTER the floor was learned quiet, and stays loud with
100+// nobody speaking. The gate opens (it cannot know better), but the watchdog
101+// must force it shut and the floor must re-learn, so this cannot run on.
102+{
103+ const frames = [...rep(0.002, 200), ...rep(0.03, 8000)]
104+ const g = new SpeechGate()
105+ let sent = 0
106+ let lastOpenAt = -1
107+ frames.forEach((rms, i) => {
108+ const e = g.push(rms)
109+ if (e === 'open') lastOpenAt = i
110+ if (g.speaking) sent++
111+ })
112+ check('a room that turns loud is eventually learned', !g.speaking)
113+ check(
114+ 'and costs exactly one watchdog window, not a repeating cycle',
115+ lastOpenAt < 250 && sent <= 1600,
116+ `${(sent * 0.04).toFixed(0)} s sent over ${(frames.length * 0.04).toFixed(0)} s; last opening at frame ${lastOpenAt}`
117+ )
118+}
119+
120+// 9. Cost: a realistic conversation, one participant's channel. They talk in
121+// 5 s bursts about a quarter of the time and listen the rest.
122+{
123+ const frames = []
124+ for (let turn = 0; turn < 24; turn++) {
125+ frames.push(...rep(0.0015, 375)) // 15 s listening
126+ for (let w = 0; w < 25; w++) {
127+ // 5 s of speech is not a plateau: syllables and gaps between words.
128+ frames.push(...rep(0.04, 3), ...rep(0.008, 2))
129+ }
130+ }
131+ const g = new SpeechGate()
132+ let sent = 0
133+ frames.forEach(rms => {
134+ g.push(rms)
135+ if (g.speaking) sent++
136+ })
137+ const talkFraction = 125 / 500
138+ const sentFraction = sent / frames.length
139+ check(
140+ 'gating tracks actual speech',
141+ sentFraction > talkFraction && sentFraction < talkFraction * 1.6,
142+ `${(sentFraction * 100).toFixed(0)}% sent vs ${(talkFraction * 100).toFixed(0)}% spoken`
143+ )
144+}
145+
146+console.log(failed === 0 ? '\nall checks passed' : `\n${failed} FAILED`)
147+process.exit(failed === 0 ? 0 : 1)
src/transcribe/store.test.mjsadded+174−0View file
@@ -0,0 +1,174 @@
1+// Checks on the transcript store in store.ts — the part that has to not lose
2+// anything, since a transcript is the one thing here meant to survive the
3+// call. It touches only localStorage and timers, both shimmed below, so it
4+// runs under node once store.ts is bundled:
5+//
6+// npx esbuild src/transcribe/store.ts --format=esm --outfile=/tmp/store.mjs
7+// node src/transcribe/store.test.mjs /tmp/store.mjs
8+
9+const backing = new Map()
10+globalThis.localStorage = {
11+ getItem: k => (backing.has(k) ? backing.get(k) : null),
12+ setItem: (k, v) => backing.set(k, String(v)),
13+ removeItem: k => backing.delete(k)
14+}
15+// Saves are debounced through window.setTimeout; the tests drive flush()
16+// directly rather than waiting on wall-clock time.
17+globalThis.window = {setTimeout: setTimeout, clearTimeout: clearTimeout}
18+
19+const {TranscriptStore, formatTranscript} = await import(
20+ process.argv[2] ?? '/tmp/store.mjs'
21+)
22+
23+let failed = 0
24+const check = (label, ok, detail = '') => {
25+ if (!ok) failed++
26+ console.log(`${ok ? 'ok ' : 'FAIL'} ${label}${detail ? ` — ${detail}` : ''}`)
27+}
28+
29+const open = room => new TranscriptStore(room, () => {})
30+
31+// 1. A transcript comes back in the room it was made in, and only there.
32+{
33+ const a = open('standup')
34+ a.append('peer1', 'Ada', 'the build is green')
35+ a.append('peer2', 'Grace', 'ship it')
36+ a.flush()
37+
38+ const reopened = open('standup')
39+ check('a transcript survives a rejoin', reopened.items.length === 2)
40+ check(
41+ 'with speakers and text intact',
42+ reopened.items[0].name === 'Ada' &&
43+ reopened.items[0].text === 'the build is green' &&
44+ reopened.items[1].name === 'Grace',
45+ reopened.items.map(i => `${i.name}: ${i.text}`).join(' | ')
46+ )
47+ check('a different room is a different transcript', open('retro').items.length === 0)
48+}
49+
50+// 2. Consecutive results from one speaker join into a paragraph; a change of
51+// speaker starts a new one.
52+{
53+ const s = open('merge')
54+ s.append('peer1', 'Ada', 'one')
55+ s.append('peer1', 'Ada', 'two')
56+ s.append('peer2', 'Grace', 'three')
57+ check(
58+ 'a speaker’s results merge into a paragraph',
59+ s.items.length === 2 && s.items[0].text === 'one two',
60+ s.items.map(i => i.text).join(' | ')
61+ )
62+}
63+
64+// 3. A restored transcript is never merged into: a new sitting starts its own
65+// paragraph even if the last line was from the same person.
66+{
67+ const s = open('sitting')
68+ s.append('peer1', 'Ada', 'yesterday')
69+ s.flush()
70+ const next = open('sitting')
71+ next.append('peer1', 'Ada', 'today')
72+ check(
73+ 'a new sitting starts a new paragraph',
74+ next.items.length === 2,
75+ next.items.map(i => i.text).join(' | ')
76+ )
77+}
78+
79+// 4. Sequence numbers are unique after a reload — they are React keys.
80+{
81+ const s = open('keys')
82+ for (let i = 0; i < 5; i++) s.append(`p${i}`, `N${i}`, `line ${i}`)
83+ s.flush()
84+ const next = open('keys')
85+ next.append('px', 'Nx', 'fresh')
86+ const seqs = next.items.map(i => i.seq)
87+ check('sequence numbers stay unique', new Set(seqs).size === seqs.length)
88+}
89+
90+// 5. Audio seconds accumulate across sittings, since the bill does.
91+{
92+ const s = open('cost')
93+ s.addAudioSeconds(30)
94+ s.append('p', 'N', 'x')
95+ s.flush()
96+ const next = open('cost')
97+ check('audio seconds carry over', next.audioSeconds === 30, `${next.audioSeconds}`)
98+ next.addAudioSeconds(15)
99+ check('and keep accumulating', next.audioSeconds === 45)
100+}
101+
102+// 6. A long-running room must not grow without bound on disk. Trimming keeps
103+// the recent end, which is the part worth having.
104+{
105+ const s = open('marathon')
106+ const line = 'x'.repeat(1000)
107+ for (let i = 0; i < 400; i++) s.append('p', 'N', `${i} ${line}`)
108+ s.flush()
109+ const stored = backing.get('commonroom:transcript:marathon')
110+ const next = open('marathon')
111+ check(
112+ 'storage stays bounded',
113+ stored.length < 300_000,
114+ `${(stored.length / 1000).toFixed(0)} kB for ${(400 * 1000) / 1000} kB of text`
115+ )
116+ check('and keeps the most recent lines', next.items.at(-1).text.startsWith('399 '))
117+ check('dropping the oldest', !next.items[0].text.startsWith('0 '))
118+}
119+
120+// 7. Clearing removes it from disk too, not just from the screen.
121+{
122+ const s = open('gone')
123+ s.append('p', 'N', 'secret')
124+ s.flush()
125+ check('written before clearing', backing.has('commonroom:transcript:gone'))
126+ s.clear()
127+ check('clear empties the panel', s.items.length === 0)
128+ check('and the stored copy', !backing.has('commonroom:transcript:gone'))
129+ check('and does not come back on rejoin', open('gone').items.length === 0)
130+ check('and resets the cost readout', open('gone').audioSeconds === 0)
131+}
132+
133+// 8. Corrupt or foreign storage must not break entering a room.
134+{
135+ backing.set('commonroom:transcript:bad', 'not json at all')
136+ check('unparseable storage is ignored', open('bad').items.length === 0)
137+ backing.set('commonroom:transcript:bad2', JSON.stringify({v: 99, items: [1, 2]}))
138+ check('a future version is ignored', open('bad2').items.length === 0)
139+ backing.set(
140+ 'commonroom:transcript:bad3',
141+ JSON.stringify({v: 1, items: [{t: 'kept', ts: 1}, {nope: true}, null]})
142+ )
143+ check('malformed entries are skipped, good ones kept', open('bad3').items.length === 1)
144+}
145+
146+// 9. Storage that throws (private mode, quota) must not take the call down.
147+{
148+ const good = globalThis.localStorage.setItem
149+ globalThis.localStorage.setItem = () => {
150+ throw new Error('QuotaExceededError')
151+ }
152+ let threw = false
153+ try {
154+ const s = open('quota')
155+ s.append('p', 'N', 'still on screen')
156+ s.flush()
157+ check('a failing write leaves the transcript on screen', s.items.length === 1)
158+ } catch {
159+ threw = true
160+ }
161+ globalThis.localStorage.setItem = good
162+ check('a failing write does not throw', !threw)
163+}
164+
165+// 10. The exported text is what lands in the clipboard and the .txt file.
166+{
167+ const s = open('format')
168+ s.append('p1', 'Ada', 'hello')
169+ const out = formatTranscript(s.items)
170+ check('formatted output names the speaker', /\] Ada: hello$/.test(out), out)
171+}
172+
173+console.log(failed === 0 ? '\nall checks passed' : `\n${failed} FAILED`)
174+process.exit(failed === 0 ? 0 : 1)
src/transcribe/store.tsadded+200−0View file
@@ -0,0 +1,200 @@
1+// The transcript itself, and its persistence.
2+//
3+// Unlike the chat, which is deliberately ephemeral, a transcript is something
4+// people come back to, so it outlives the call: it is kept in localStorage
5+// under the room it belongs to and restored when you next enter that room.
6+// This is per browser and never leaves it — the transcript is not shared with
7+// the other participants (see the transcription section of network.ts), so
8+// what is stored here is only ever what this browser transcribed.
9+//
10+// Because it does persist, it can be cleared, and the panel asks before doing
11+// so. Note that transcripts of meetings sitting in localStorage in plain text
12+// is a real consideration on a shared machine; the Clear button is the answer,
13+// and the panel says as much.
14+
15+const KEY_PREFIX = 'commonroom:transcript:'
16+const VERSION = 1
17+
18+/** Hard caps. The first bounds memory in a marathon call, the second bounds
19+ * what is written back, since localStorage is a small shared budget. */
20+const ITEM_CAP = 2000
21+const MAX_STORED_CHARS = 200_000
22+
23+/** Consecutive results from one speaker join into a paragraph if they arrive
24+ * within this long of each other. */
25+const MERGE_GAP_MS = 15_000
26+const MERGE_MAX_CHARS = 700
27+
28+/** Writes are coalesced: speech produces results every couple of seconds and
29+ * the whole transcript is rewritten each time. */
30+const SAVE_DEBOUNCE_MS = 2000
31+
32+export interface TranscriptItem {
33+ /** Monotonic per-session sequence number; stable React key. */
34+ seq: number
35+ /** Whose speech this is (peer ID, or the local peer's own ID). */
36+ peerId: string
37+ name: string
38+ text: string
39+ /** Local time the paragraph started (epoch ms). */
40+ time: number
41+}
42+
43+interface StoredItem {
44+ p: string
45+ n: string
46+ t: string
47+ ts: number
48+}
49+
50+export class TranscriptStore {
51+ readonly items: TranscriptItem[] = []
52+ private seq = 0
53+ private key: string
54+ private saveTimer: number | null = null
55+ private lastAppendAt = 0
56+ private lastAppendId: string | null = null
57+ /** Audio sent to the transcription service for this room, in seconds. */
58+ private seconds = 0
59+
60+ constructor(
61+ roomId: string,
62+ private onChange: () => void
63+ ) {
64+ this.key = KEY_PREFIX + roomId
65+ this.load()
66+ }
67+
68+ get audioSeconds(): number {
69+ return this.seconds
70+ }
71+
72+ addAudioSeconds(s: number) {
73+ this.seconds += s
74+ }
75+
76+ append(peerId: string, name: string, text: string) {
77+ const t = text.trim()
78+ if (!t) return
79+ const now = Date.now()
80+ const last = this.items[this.items.length - 1]
81+ if (
82+ last &&
83+ this.lastAppendId === peerId &&
84+ now - this.lastAppendAt < MERGE_GAP_MS &&
85+ last.text.length < MERGE_MAX_CHARS
86+ ) {
87+ this.items[this.items.length - 1] = {
88+ ...last,
89+ name: name || last.name,
90+ text: `${last.text} ${t}`
91+ }
92+ } else {
93+ this.items.push({seq: this.seq++, peerId, name, text: t, time: now})
94+ if (this.items.length > ITEM_CAP) {
95+ this.items.splice(0, this.items.length - ITEM_CAP)
96+ }
97+ }
98+ this.lastAppendAt = now
99+ this.lastAppendId = peerId
100+ this.scheduleSave()
101+ this.onChange()
102+ }
103+
104+ clear() {
105+ this.items.length = 0
106+ this.seconds = 0
107+ this.lastAppendId = null
108+ if (this.saveTimer !== null) clearTimeout(this.saveTimer)
109+ this.saveTimer = null
110+ try {
111+ localStorage.removeItem(this.key)
112+ } catch {
113+ /* storage unavailable; nothing was written in the first place */
114+ }
115+ this.onChange()
116+ }
117+
118+ /** Write out any pending changes now — on leaving the room, or on unload. */
119+ flush() {
120+ if (this.saveTimer === null) return
121+ clearTimeout(this.saveTimer)
122+ this.saveTimer = null
123+ this.save()
124+ }
125+
126+ private scheduleSave() {
127+ if (this.saveTimer !== null) return
128+ this.saveTimer = window.setTimeout(() => {
129+ this.saveTimer = null
130+ this.save()
131+ }, SAVE_DEBOUNCE_MS)
132+ }
133+
134+ private save() {
135+ // Trim from the front until the payload fits: the recent end of a
136+ // transcript is the part worth keeping.
137+ let from = 0
138+ let chars = this.items.reduce((n, i) => n + i.text.length + i.name.length, 0)
139+ while (from < this.items.length && chars > MAX_STORED_CHARS) {
140+ chars -= this.items[from].text.length + this.items[from].name.length
141+ from++
142+ }
143+ const stored: StoredItem[] = this.items.slice(from).map(i => ({
144+ p: i.peerId,
145+ n: i.name,
146+ t: i.text,
147+ ts: i.time
148+ }))
149+ try {
150+ localStorage.setItem(
151+ this.key,
152+ JSON.stringify({v: VERSION, seconds: this.seconds, items: stored})
153+ )
154+ } catch {
155+ // Quota exceeded, or storage disabled. The transcript is still on
156+ // screen and can be saved to a file; failing the call over it would be
157+ // worse than losing the persistence.
158+ }
159+ }
160+
161+ private load() {
162+ let raw: string | null
163+ try {
164+ raw = localStorage.getItem(this.key)
165+ } catch {
166+ return
167+ }
168+ if (!raw) return
169+ let data: {v?: unknown; seconds?: unknown; items?: unknown}
170+ try {
171+ data = JSON.parse(raw)
172+ } catch {
173+ return
174+ }
175+ if (data.v !== VERSION || !Array.isArray(data.items)) return
176+ if (typeof data.seconds === 'number' && data.seconds >= 0) {
177+ this.seconds = data.seconds
178+ }
179+ for (const entry of data.items as StoredItem[]) {
180+ if (typeof entry?.t !== 'string' || typeof entry.ts !== 'number') continue
181+ this.items.push({
182+ seq: this.seq++,
183+ peerId: typeof entry.p === 'string' ? entry.p : '',
184+ name: typeof entry.n === 'string' ? entry.n : '',
185+ text: entry.t,
186+ time: entry.ts
187+ })
188+ }
189+ // A restored transcript never merges into: the next thing said belongs to
190+ // a new sitting, whatever the clock says.
191+ this.lastAppendId = null
192+ }
193+}
194+
195+const stamp = (t: number): string =>
196+ new Date(t).toLocaleTimeString([], {hour: '2-digit', minute: '2-digit'})
197+
198+/** The transcript as plain text, for copying or saving. */
199+export const formatTranscript = (items: TranscriptItem[]): string =>
200+ items.map(i => `[${stamp(i.time)}] ${i.name}: ${i.text}`).join('\n\n')
src/transcribe/transcriber.tsadded+166−0View file
@@ -0,0 +1,166 @@
1+// Meeting transcription: one gated capture and one Deepgram connection per
2+// participant, feeding a single time-ordered transcript.
3+//
4+// Only the person who entered an API key runs any of this. They already have
5+// every participant's audio locally (that is what a mesh call is), so one
6+// browser can transcribe the whole room. Note the trade-off: peers' audio has
7+// been through Opus at the room's bitrate by the time we see it, so it is not
8+// as clean as it would be captured at the source. The alternative — everyone
9+// transcribing their own microphone and publishing the text — would need a
10+// credential for each participant and a way to share it, and would put the
11+// authorship of every line in the hands of whoever sent it.
12+//
13+// Attribution is structural rather than inferred: each socket carries exactly
14+// one speaker, so no diarization is involved and there is nothing to
15+// misattribute.
16+
17+import {registerCaptureWorklet, SpeechCapture} from './capture'
18+import {DeepgramStream} from './deepgram'
19+import type {TranscriptStore} from './store'
20+
21+/** Preferred capture rate. Speech models work at 16 kHz, and sending 16-bit
22+ * samples at that rate is a third of the bytes of a 48 kHz stream. */
23+const PREFERRED_RATE = 16000
24+
25+export interface TranscriptSource {
26+ id: string
27+ name: string
28+ stream: MediaStream | null
29+}
30+
31+interface Pipeline {
32+ /** Source stream plus track identity: a microphone swapped in mid-call is a
33+ * new track, and the Web Audio source node does not follow the change. */
34+ key: string
35+ name: string
36+ capture: SpeechCapture
37+ stream: DeepgramStream
38+}
39+
40+export class Transcriber {
41+ private ctx: AudioContext | null = null
42+ private pipelines = new Map<string, Pipeline>()
43+ private running = false
44+
45+ constructor(
46+ readonly apiKey: string,
47+ private store: TranscriptStore,
48+ private onChange: () => void,
49+ private onFailure: (message: string, fatal: boolean) => void
50+ ) {}
51+
52+ get active(): boolean {
53+ return this.running
54+ }
55+
56+ async start(): Promise<boolean> {
57+ if (this.running) return true
58+ if (!this.ctx) {
59+ const fail = (err: unknown) => {
60+ const why = err instanceof Error ? err.message : 'audio setup failed'
61+ this.onFailure(`Transcription could not start — ${why}.`, true)
62+ return false
63+ }
64+ let ctx: AudioContext
65+ try {
66+ // Not every browser will honor an explicit rate; falling back costs
67+ // bandwidth, not correctness, since the real rate is what we declare
68+ // to Deepgram.
69+ try {
70+ ctx = new AudioContext({sampleRate: PREFERRED_RATE})
71+ } catch {
72+ ctx = new AudioContext()
73+ }
74+ } catch (err) {
75+ return fail(err)
76+ }
77+ try {
78+ await registerCaptureWorklet(ctx)
79+ } catch (err) {
80+ void ctx.close().catch(() => undefined)
81+ return fail(err)
82+ }
83+ this.ctx = ctx
84+ }
85+ // Starting transcription is a click, so the context is allowed to run;
86+ // resuming matters when a previous stop left it suspended.
87+ await this.ctx.resume().catch(() => undefined)
88+ this.running = true
89+ return true
90+ }
91+
92+ /** Reconcile the running pipelines with who is in the room. Cheap to call on
93+ * every snapshot: it only acts on what actually changed. */
94+ setSources(sources: TranscriptSource[]) {
95+ if (!this.running || !this.ctx) return
96+ const seen = new Set<string>()
97+ for (const src of sources) {
98+ const media = src.stream
99+ const track = media?.getAudioTracks()[0]
100+ if (!media || !track) continue
101+ seen.add(src.id)
102+ const key = `${media.id}:${track.id}`
103+ const existing = this.pipelines.get(src.id)
104+ if (existing) {
105+ existing.name = src.name
106+ if (existing.key === key) continue
107+ this.destroy(src.id)
108+ }
109+ this.create(src.id, src.name, key, media)
110+ }
111+ for (const id of [...this.pipelines.keys()]) {
112+ if (!seen.has(id)) this.destroy(id)
113+ }
114+ }
115+
116+ private create(id: string, name: string, key: string, media: MediaStream) {
117+ const ctx = this.ctx
118+ if (!ctx) return
119+ const stream = new DeepgramStream(this.apiKey, ctx.sampleRate, {
120+ // Look the name up late, so a rename is reflected — but fall back to the
121+ // one we had, since a result can still arrive after the speaker left and
122+ // an unattributed transcript line is worse than a stale name.
123+ transcript: text =>
124+ this.store.append(id, this.pipelines.get(id)?.name ?? name, text),
125+ sent: samples => this.store.addAudioSeconds(samples / ctx.sampleRate),
126+ failure: (message, fatal) => this.onFailure(message, fatal)
127+ })
128+ let capture: SpeechCapture
129+ try {
130+ capture = new SpeechCapture(ctx, media, {
131+ frame: pcm => stream.send(pcm),
132+ end: () => stream.finalize()
133+ })
134+ } catch {
135+ // A stream whose track vanished between the snapshot and here; the next
136+ // reconcile will pick it up again if it comes back.
137+ stream.stop()
138+ return
139+ }
140+ this.pipelines.set(id, {key, name, capture, stream})
141+ }
142+
143+ private destroy(id: string) {
144+ const p = this.pipelines.get(id)
145+ if (!p) return
146+ this.pipelines.delete(id)
147+ p.capture.close()
148+ p.stream.stop()
149+ }
150+
151+ /** Stop capturing. The sockets wind down gracefully, so a sentence that was
152+ * in flight still reaches the transcript, and the transcript itself stays. */
153+ stop() {
154+ this.running = false
155+ for (const id of [...this.pipelines.keys()]) this.destroy(id)
156+ void this.ctx?.suspend().catch(() => undefined)
157+ this.onChange()
158+ }
159+
160+ dispose() {
161+ this.stop()
162+ const ctx = this.ctx
163+ this.ctx = null
164+ if (ctx) void ctx.close().catch(() => undefined)
165+ }
166+}
tsconfig.tsbuildinfomodified+1−1View file
@@ -1 +1 @@
1-{"root":["./src/App.tsx","./src/main.tsx","./src/useNetwork.ts","./src/vite-env.d.ts","./src/p2p/identity.ts","./src/p2p/network.ts","./src/p2p/nostr.ts","./src/p2p/peer.ts","./src/p2p/settings.ts","./src/p2p/turn.ts"],"version":"5.9.3"}
1+{"root":["./src/App.tsx","./src/main.tsx","./src/useNetwork.ts","./src/vite-env.d.ts","./src/p2p/identity.ts","./src/p2p/network.ts","./src/p2p/nostr.ts","./src/p2p/peer.ts","./src/p2p/settings.ts","./src/p2p/turn.ts","./src/transcribe/capture.ts","./src/transcribe/deepgram.ts","./src/transcribe/store.ts","./src/transcribe/transcriber.ts"],"version":"5.9.3"}