1# CLAUDE.md
3Tips for future agents working in this repo — a Node CLI that joins a
4[commonroom](https://github.com/concept-collection/commonroom) call as a
5muted, visible participant and records every other participant's audio (one
6WAV per participant) plus the room chat. Read commonroom's CLAUDE.md first:
7this tool speaks its protocol verbatim, and the protocol is documented there.
9## Architecture
11```
12src/
13 identity.ts ported from commonroom; EPHEMERAL keypair (no localStorage)
14 nostr.ts ported near-verbatim (Node >= 22 global WebSocket); close() added
15 peer.ts ported, adapted to @roamhq/wrtc; receive-only media (see below)
16 wav.ts incremental WAV writer (buffers ~1 s, re-patches header sizes)
17 recorder.ts the heart: presence, mesh, control channel, audio sinks, files
18 render.ts shared transcript timeline: turn merging, md/json rendering
19 transcribe.ts `transcribe` subcommand: per-WAV ASR + merged transcript.md
20 livetranscribe.ts record --transcribe: persistent faster-whisper helper,
21 silence-cut chunking, live transcript re-rendering
22 cli.ts record/transcribe subcommands, signal handling, log lines
23 test/
24 speaker.ts synthetic participant: sine tone + one chat message
25 loopback.ts test: recorder + speaker in a random room -> tone + chat
26 transcribe-test.ts test: fabricated 2-speaker recording dir (JFK sample)
27 -> transcript ordering + words; uses --model tiny
28```
30## Key design decisions
32- **Protocol-identical participant.** Same announcements, per-peer signaling
33 topics, deterministic initiator (smaller peer ID), control-channel messages
34 (hello/mute/chat/bye), stalled-connection retry, room-full handling, and the
35 soft cap of 8 — the recorder counts toward it. Don't invent protocol; if the
36 browser client changes, port the change.
37- **Receive-only media, but symmetric-looking.** Outgoing tracks are wrtc
38 `RTCAudioSource`/`RTCVideoSource` placeholders that never produce data (=
39 a fully muted participant). The VIDEO m-line is negotiated `sendonly` from
40 our side (as initiator via `addTransceiver(track, {direction:'sendonly'})`;
41 as answerer by flipping the transceiver direction after
42 setRemoteDescription) so browsers never send us video — saves bandwidth and
43 decode CPU. Audio stays sendrecv.
44- **Recording gates on first non-zero frame.** Before the first RTP packet,
45 `RTCAudioSink` delivers all-zero frames at a PROVISIONAL sample rate (16 kHz
46 observed), then switches to the real one (48 kHz) — opening the file eagerly
47 yields junk stub segments. A never-unmuting participant produces no file. A
48 mid-stream format change (rare) closes the segment and starts a new one.
49- **Wall-clock silence padding.** If the sink stalls > 1 s (network gap, DTX),
50 silence is inserted so sample position keeps tracking elapsed time — the
51 manifest's segment `startedAt` plus the file offset IS the meeting timeline.
52- **connectionState flaps.** wrtc can pass through 'connected' several times
53 while ICE settles; the connect handler must be idempotent or hello/notice
54 get re-sent per flap.
55- **Bye cooldown (3 s).** An announcement published just before a peer's bye
56 can arrive just after it and would trigger an instant reconnect (and a stub
57 recording); after a bye we ignore that peer's announcements briefly.
58- **The recording dir is also the agent interface.** `inbox/` is polled every
59 500 ms; a file's content is broadcast as a chat message from the recorder
60 (then the file is deleted; dotfiles/*.tmp skipped, 300 ms mtime settle,
61 2000-char cap) and logged everywhere a received chat would be. `AGENT.md`
62 (template at the bottom of recorder.ts) is written at start with
63 instructions for a monitoring AI agent.
64- **Every exit path MUST end in `process.exit()`.** @roamhq/wrtc segfaults in
65 its static destructors on a natural process exit whenever nonstandard
66 media sources exist. The CLI, the speaker, and any future script that
67 touches wrtc must exit explicitly.
68- **Crash-safe outputs.** events.jsonl and chat.txt are appended per event;
69 WAVs flush (with header re-patch) about once a second; manifest.json is
70 written atomically (tmp + rename) at segment boundaries and every 30 s.
71- **Transcription needs no alignment step.** The silence padding means an ASR
72 timestamp within a segment plus the manifest `startedAt` is the wall-clock
73 time; transcribe.ts just merges utterances with chat/join/left events and
74 groups adjacent same-speaker utterances (< 3 s gap) into turns. ASR engines
75 are probed (faster-whisper via an embedded python3 stdin script — VAD on,
76 which also skips the padded silence — then whisper-cli, then whisper);
77 raw ASR is cached per WAV in `<dir>/asr/`. cli.ts imports recorder.js
78 LAZILY so transcribe works where the wrtc native module doesn't load.
79- **Live transcription (record --transcribe) never blocks recording.** One
80 persistent python process (faster-whisper only) serves requests over
81 line-JSON stdio; it reads raw flushed frame ranges straight from the
82 growing WAVs (header bypassed). Chunks are cut at >= 300 ms of quiet
83 (RMS < 300) tracked from the live sample stream — never mid-word — with a
84 30 s force-cut for unbroken speech; all-quiet chunks skip ASR entirely.
85 Constants are tuned for ~8 s typical end-of-utterance-to-text latency
86 (3 s tick, 2 s min chunk, 0.5 s render debounce).
87 Results re-render transcript.md live and are persisted to asr/*.json in the
88 offline format, so `transcribe` can re-render or upgrade models later. If
89 the helper dies, it logs once and recording continues.
90- **Spawn the persistent helper with `python3 -c <script>`**, never
91 `python3 -` + script on stdin: `-` reads stdin to EOF before executing, so
92 a process that keeps stdin open for requests never starts. (The one-shot
93 offline helper uses stdin+end() and is fine.) In the helper, read requests
94 with `sys.stdin.readline()`, not `for line in sys.stdin` (read-ahead
95 buffering sits on complete lines).
97## Testing
99`npm run build && npm run test:loopback` — full end-to-end over the real
100public relays (needs network): asserts the recorded WAV contains the 440 Hz
101tone (RMS + zero-crossing rate) and the chat message landed exactly once.
102`npm run test:transcribe` — real-speech transcription test (downloads the
103whisper.cpp JFK sample + the tiny model on first run). `npm run test:live` —
104end-to-end LIVE transcription: a speaker streams the JFK WAV into a room
105(`speaker.js --wav`; playback and leave countdown start at first connect) and
106the test asserts the transcript grew while still recording.
107Segfault-at-exit in a child process = some path bypassed `process.exit()`.
108For manual testing against real browsers, record a room and join it at
109https://concept-collection.github.io/commonroom/ — let the user do
110multi-person tests; don't try to automate browser media.