Record computer audio alongside the microphone
The server captures the default output's PipeWire/PulseAudio monitor with ffmpeg while a take records, and aligns it with the browser's recordings by wall-clock time using the recorder's start, pause, and resume events. It is stored per take as system.webm, played in the preview, and mixed into the export at an adjustable level.
13 changed files+313−22
README.mdmodified+5−1View file
@@ -9,6 +9,7 @@ We wrote it for a specific workflow: explaining or demonstrating something in a
99 - Node.js 22.18 or later (the server runs TypeScript directly through Node's type stripping)
1010 - ffmpeg and ffprobe on the `PATH`
1111 - Google Chrome or Chromium (recommended; Firefox also works for recording)
12+- PipeWire (with its PulseAudio compatibility, as on current GNOME desktops) or PulseAudio, to record computer audio
1213
1314 ## Running
1415
@@ -35,6 +36,8 @@ The library shows every video with its thumbnail, duration, and description. Dra
3536
3637 **Recording.** Choose any combination of camera, microphone, and screen, then start recording. The screen picker is GNOME's own portal dialog, which lets you choose a monitor or a single window. A countdown precedes the recording, and it can be paused and resumed. The *Teleprompter window* button opens the notes in a separate window with large text, optional auto-scroll, and start/pause/stop controls, so it can sit on a second monitor or outside the captured window. Recorded data is uploaded to the server once per second while recording, so a browser crash loses at most about a second; an interrupted take appears on the video page with a *Recover* button.
3738
39+**Computer audio.** With *Computer audio* checked, Takes also records what the computer plays through its default output (for example, the sound of a demo), in addition to the microphone. Unlike the microphone, this is not captured by the browser: the server records the PipeWire or PulseAudio monitor of the default output with ffmpeg, so it works in any browser and without sharing the screen. The two recordings are aligned by wall-clock time, which the browser and the server share since they run on the same machine. ffmpeg reports when its first sample was captured, the recorder reports when it started, paused, and resumed, and when the take is saved the capture is cut to the same intervals and stored as `system.webm`. In a test with a synthetic capture, the cut points were within 15 ms of where they should be. Computer audio is kept separate from the microphone: it is mixed in during playback and export, with its own level on the *Layout & export* tab, and it is not transcribed. Use headphones when recording both, since otherwise the microphone also picks up the sound from the speakers.
40+
3841 **Editing.** Each video has a timeline made of segments of its takes. A new take is appended to the end of the timeline, and a take can be added again from the *Takes* list. The editor supports splitting at the playhead (S), marking a range (I and O) and cutting it (X), trimming the current segment to the playhead, removing and reordering segments, and undo/redo. The timeline shows the audio waveform, and a zoom slider helps with precise cuts. Editing is non-destructive: the raw takes are never modified.
3942
4043 **Transcripts.** With a Deepgram API key (set it on the Settings page), each take's audio is transcribed after recording. The *Transcript* tab shows the words that remain in the timeline. Clicking a word seeks to it, and shift-clicking a second word selects the range between them, which can then be cut. This is often the quickest way to remove a false start or a filler word.
@@ -59,6 +62,7 @@ The library shows every video with its thumbnail, duration, and description. Dra
5962 <title>.mp4 latest export
6063 takes/<take-id>/
6164 screen.webm camera.webm raw recordings (mic audio is in camera.webm when there is a camera)
65+ system.webm computer audio, aligned with the recordings, if requested
6266 cutout.webm camera with background removed (VP9 + alpha), if requested
6367 peaks.json waveform data
6468 transcript.json Deepgram words with timestamps
@@ -70,7 +74,7 @@ Everything is plain files, so videos can be backed up, synced, or inspected with
7074
7175 - Screen and camera are recorded by two independent MediaRecorders started at the same instant. In our tests they stay aligned to within a frame, but we have not measured drift over recordings longer than about 20 minutes.
7276 - Screen capture on Wayland depends on the browser's PipeWire support. Recent Chrome enables it by default; if the picker does not appear or the capture is black, check `chrome://flags/#enable-webrtc-pipewire-capturer`.
73-- Only the microphone is recorded; system audio (for example, sound from a demo) is not captured.
77+- Computer audio is taken from the default output as a whole, so notification sounds and any other audio playing at the time are recorded too; there is no per-application capture. The alignment depends on the browser starting its recorders promptly when asked, which we have not measured directly.
7478 - The camera layout and background are set per video, not per segment; crops are set per take.
7579 - A crop is stored relative to the frame, so if a window recording changes size partway through, the crop follows the new frame rather than the same content.
7680 - Background removal on the CPU runs well below real time on a laptop (see above for running it on a remote GPU); the matting quality also degrades with a cluttered background, poor lighting, or objects held close to the body, which the model may partly treat as background.
server/export.tsmodified+16−7View file
@@ -7,7 +7,7 @@
77 import fs from 'node:fs/promises';
88 import path from 'node:path';
99 import type { Crop, ExportJob, Video } from '../shared/types.ts';
10-import { cropOf, exportSignature } from '../shared/types.ts';
10+import { cropOf, exportSignature, systemVolumeOf } from '../shared/types.ts';
1111 import { FPS, computeGeometry } from '../shared/layout.ts';
1212 import { timelineDuration } from '../shared/timeline.ts';
1313 import { fileSize, run } from './media.ts';
@@ -56,6 +56,8 @@ export function buildExportArgs(v: Video, outFile: string): string[] {
5656 if (bgMode !== 'none') idx.cutout = addInput(path.join(dir, 'cutout.webm'), seg.start, D, 'libvpx-vp9');
5757 if (g.camera && (bgMode === 'none' || bgMode === 'blur')) idx.camera = addInput(path.join(dir, 'camera.webm'), seg.start, D);
5858 if (take.audio && idx[take.audio] === undefined) idx[take.audio] = addInput(path.join(dir, `${take.audio}.webm`), seg.start, D);
59+ const sysVol = systemVolumeOf(v.output);
60+ if (take.systemAudio && sysVol > 0) idx.system = addInput(path.join(dir, 'system.webm'), seg.start, D);
5961
6062 filters.push(`color=c=black:s=${g.W}x${g.H}:r=${FPS}:d=${f3(D)}[bg${i}]`);
6163 let cur = `bg${i}`;
@@ -99,15 +101,22 @@ export function buildExportArgs(v: Video, outFile: string): string[] {
99101 }
100102 filters.push(`[${cur}]format=yuv420p[v${i}]`);
101103
104+ // Microphone and computer audio, each trimmed/padded to the segment length, summed, with short fades against clicks.
102105 const fade = Math.min(0.015, D / 4);
106+ const fit = `aresample=48000,aformat=sample_fmts=fltp:channel_layouts=stereo,apad,atrim=0:${f3(D)},asetpts=PTS-STARTPTS`;
107+ const fades = `afade=t=in:d=${f3(fade)},afade=t=out:st=${f3(D - fade)}:d=${f3(fade)}`;
108+ const sources: string[] = [];
103109 if (take.audio) {
104- filters.push(
105- `[${idx[take.audio]}:a]aresample=48000,aformat=sample_fmts=fltp:channel_layouts=stereo,apad,atrim=0:${f3(D)},asetpts=PTS-STARTPTS,` +
106- `afade=t=in:d=${f3(fade)},afade=t=out:st=${f3(D - fade)}:d=${f3(fade)}[a${i}]`,
107- );
108- } else {
109- filters.push(`anullsrc=r=48000:cl=stereo,atrim=0:${f3(D)}[a${i}]`);
110+ filters.push(`[${idx[take.audio]}:a]${fit}[am${i}]`);
111+ sources.push(`[am${i}]`);
112+ }
113+ if (idx.system !== undefined) {
114+ filters.push(`[${idx.system}:a]${fit},volume=${sysVol.toFixed(3)}[as${i}]`);
115+ sources.push(`[as${i}]`);
110116 }
117+ if (sources.length === 2) filters.push(`${sources.join('')}amix=inputs=2:duration=first:normalize=0,${fades}[a${i}]`);
118+ else if (sources.length === 1) filters.push(`${sources[0]}${fades}[a${i}]`);
119+ else filters.push(`anullsrc=r=48000:cl=stereo,atrim=0:${f3(D)}[a${i}]`);
111120 concat += `[v${i}][a${i}]`;
112121 });
113122
server/index.tsmodified+14−1View file
@@ -14,6 +14,7 @@ import { isFullFrame } from '../shared/types.ts';
1414 import { getExportJob, startExport } from './export.ts';
1515 import { HttpError, checkId, createVideo, getConfig, getVideo, initStore, libraryDir, listVideos, setOrder, trashVideo, updateConfig, updateVideo, videoDir } from './store.ts';
1616 import { appendChunk, createTake, deleteTake, finalizeTake, setThumbnail } from './takes.ts';
17+import { systemCaptureEvent } from './sysaudio.ts';
1718 import { transcribeTake } from './transcribe.ts';
1819 import { requestMattes, withMatteStatus } from './matte.ts';
1920
@@ -119,7 +120,11 @@ app.patch('/api/videos/:id', json, async (req, res) => {
119120 throw new HttpError(400, 'Invalid background settings');
120121 }
121122 }
122- if (b.output) v.output = { ...v.output, ...b.output };
123+ if (b.output) {
124+ v.output = { ...v.output, ...b.output };
125+ const sv = v.output.systemVolume;
126+ if (sv !== undefined && !(typeof sv === 'number' && sv >= 0 && sv <= 1)) throw new HttpError(400, 'Invalid computer audio level');
127+ }
123128 if (b.crops !== undefined) v.crops = parseCrops(b.crops, new Set(v.takes.map((t) => t.id)));
124129 if (Array.isArray(b.timeline)) {
125130 const takes = new Map(v.takes.map((t) => [t.id, t]));
@@ -166,6 +171,14 @@ app.post('/api/videos/:id/takes/:takeId/chunk', raw, async (req, res) => {
166171 res.json({ ok: true });
167172 });
168173
174+// Recorder start/pause/resume times (for aligning computer audio), or a keep-alive with no type.
175+app.post('/api/videos/:id/takes/:takeId/events', json, async (req, res) => {
176+ const b = req.body ?? {};
177+ const ev = ['start', 'pause', 'resume'].includes(b.type) && Number.isFinite(b.at) ? { type: b.type, at: b.at } : null;
178+ await systemCaptureEvent(checkId(p(req, 'id')), checkId(p(req, 'takeId')), ev);
179+ res.json({ ok: true });
180+});
181+
169182 app.post('/api/videos/:id/takes/:takeId/finalize', async (req, res) => {
170183 res.json(withMatteStatus(await finalizeTake(p(req, 'id'), p(req, 'takeId'))));
171184 });
server/sysaudio.tsadded+163−0View file
@@ -0,0 +1,163 @@
1+// Computer ("system") audio: what the speakers play, captured by ffmpeg from the PipeWire/PulseAudio
2+// monitor of the default output while a take records.
3+//
4+// The browser's recordings and this capture are separate, so they are aligned by wall-clock time,
5+// which both share since they run on the same machine. ffmpeg reports the wall-clock time of the first
6+// captured sample, and the browser reports when its recorders started, paused, and resumed. When
7+// the take is finalized, the capture is cut to the same intervals and written as system.webm.
8+
9+import { spawn, type ChildProcess } from 'node:child_process';
10+import fs from 'node:fs/promises';
11+import path from 'node:path';
12+import { run } from './media.ts';
13+import { exists, takeDir, writeJsonAtomic } from './store.ts';
14+
15+export const SYSTEM_SOURCE = '@DEFAULT_MONITOR@';
16+// Without an event from the recorder for this long (it sends one every few seconds), the capture is stopped.
17+const IDLE_MS = 60_000;
18+
19+export type RecEvent = { type: 'start' | 'pause' | 'resume'; at: number }; // at: ms since the epoch
20+
21+// Persisted next to the capture, so that an interrupted take can still be aligned when recovered.
22+interface Timing {
23+ audioStart: number; // wall-clock time of the first captured sample, ms since the epoch
24+ events: RecEvent[];
25+}
26+
27+interface Capture {
28+ proc: ChildProcess;
29+ exited: Promise<void>;
30+ lastSeen: number;
31+ watchdog: NodeJS.Timeout;
32+}
33+
34+const captures = new Map<string, Capture>();
35+const partFile = (videoId: string, takeId: string) => path.join(takeDir(videoId, takeId), 'system.part.mka');
36+const timingFile = (videoId: string, takeId: string) => path.join(takeDir(videoId, takeId), 'system-timing.json');
37+
38+async function readTiming(videoId: string, takeId: string): Promise<Timing | null> {
39+ try {
40+ return JSON.parse(await fs.readFile(timingFile(videoId, takeId), 'utf8')) as Timing;
41+ } catch {
42+ return null;
43+ }
44+}
45+
46+// Starts capturing and resolves once the first samples have arrived (so the capture always
47+// begins before the recording does), or rejects if ffmpeg cannot open the audio server.
48+export async function startSystemCapture(videoId: string, takeId: string): Promise<void> {
49+ const key = `${videoId}/${takeId}`;
50+ if (captures.has(key)) return;
51+ const args = ['-hide_banner', '-nostats', '-f', 'pulse', '-i', SYSTEM_SOURCE, '-ac', '2', '-ar', '48000', '-c:a', 'libopus', '-b:a', '128k', '-f', 'matroska', '-y', partFile(videoId, takeId)];
52+ const proc = spawn('ffmpeg', args, { stdio: ['pipe', 'ignore', 'pipe'] });
53+ let stderr = '';
54+ const exited = new Promise<void>((resolve) => proc.on('close', () => resolve()));
55+ const audioStart = await new Promise<number>((resolve, reject) => {
56+ const fail = (msg: string) => {
57+ proc.kill('SIGKILL');
58+ reject(new Error(`Could not capture computer audio: ${msg}`));
59+ };
60+ const timer = setTimeout(() => fail('no audio arrived from the sound server within 5 s'), 5000);
61+ proc.stderr!.setEncoding('utf8');
62+ proc.stderr!.on('data', (d: string) => {
63+ stderr = (stderr + d).slice(-20_000);
64+ const m = /Input #0, pulse.*\n.*start: ([\d.]+)/.exec(stderr);
65+ if (m) {
66+ clearTimeout(timer);
67+ resolve(parseFloat(m[1]) * 1000);
68+ }
69+ });
70+ proc.on('error', (e) => {
71+ clearTimeout(timer);
72+ fail(e.message);
73+ });
74+ proc.on('close', (code) => {
75+ clearTimeout(timer);
76+ fail(`ffmpeg exited with code ${code}: ${stderr.trim().split('\n').slice(-3).join(' ')}`);
77+ });
78+ });
79+ await writeJsonAtomic(timingFile(videoId, takeId), { audioStart, events: [] } satisfies Timing);
80+ const cap = { proc, exited, lastSeen: Date.now() } as Capture;
81+ cap.watchdog = setInterval(() => {
82+ if (Date.now() - cap.lastSeen > IDLE_MS) {
83+ console.warn(`system audio: no activity for take ${takeId}; stopping the capture`);
84+ stopSystemCapture(videoId, takeId);
85+ }
86+ }, 5000);
87+ captures.set(key, cap);
88+}
89+
90+// Records a recorder event (or, with no event, just that the recorder is still alive).
91+export async function systemCaptureEvent(videoId: string, takeId: string, ev: RecEvent | null) {
92+ const cap = captures.get(`${videoId}/${takeId}`);
93+ if (cap) cap.lastSeen = Date.now();
94+ if (!ev) return;
95+ const t = await readTiming(videoId, takeId);
96+ if (!t) return;
97+ t.events.push(ev);
98+ await writeJsonAtomic(timingFile(videoId, takeId), t);
99+}
100+
101+export async function stopSystemCapture(videoId: string, takeId: string) {
102+ const key = `${videoId}/${takeId}`;
103+ const cap = captures.get(key);
104+ if (!cap) return;
105+ captures.delete(key);
106+ clearInterval(cap.watchdog);
107+ // 'q' lets ffmpeg finish the file cleanly; kill it if it does not exit promptly.
108+ cap.proc.stdin?.end('q');
109+ const t = setTimeout(() => cap.proc.kill('SIGKILL'), 5000);
110+ await cap.exited;
111+ clearTimeout(t);
112+}
113+
114+// The parts of the capture that correspond to the recording, in capture time (seconds); a negative
115+// start means the recording began before the capture did. The last interval is open-ended.
116+export function keptIntervals(t: Timing): [number, number | null][] {
117+ const out: [number, number | null][] = [];
118+ let open: number | null = null;
119+ for (const ev of t.events) {
120+ const s = (ev.at - t.audioStart) / 1000;
121+ if ((ev.type === 'start' || ev.type === 'resume') && open === null) open = s;
122+ else if (ev.type === 'pause' && open !== null) {
123+ out.push([open, s]);
124+ open = null;
125+ }
126+ }
127+ if (open !== null) out.push([open, null]);
128+ return out;
129+}
130+
131+// Stops the capture and writes system.webm, aligned with the take and exactly `duration` long.
132+// Returns false if there is nothing usable (e.g. the capture never started).
133+export async function finalizeSystemAudio(videoId: string, takeId: string, duration: number): Promise<boolean> {
134+ await stopSystemCapture(videoId, takeId);
135+ const part = partFile(videoId, takeId);
136+ const timing = await readTiming(videoId, takeId);
137+ if (!(await exists(part)) || !timing) return false;
138+ const intervals = keptIntervals(timing);
139+ if (intervals.length === 0) return false;
140+ const f = (x: number) => x.toFixed(3);
141+ const filters = intervals.map(([a, b], i) => {
142+ const trim = `atrim=start=${f(Math.max(0, a))}${b !== null ? `:end=${f(Math.max(0, b))}` : ''}`;
143+ const lead = a < 0 ? `,adelay=${Math.round(-a * 1000)}:all=1` : '';
144+ return `[0:a]${trim},asetpts=PTS-STARTPTS${lead}[i${i}]`;
145+ });
146+ filters.push(`${intervals.map((_, i) => `[i${i}]`).join('')}concat=n=${intervals.length}:v=0:a=1,apad,atrim=0:${f(duration)}[out]`);
147+ const out = path.join(takeDir(videoId, takeId), 'system.webm');
148+ await run('ffmpeg', ['-y', '-v', 'error', '-i', part, '-filter_complex', filters.join(';'), '-map', '[out]', '-c:a', 'libopus', '-b:a', '128k', out]);
149+ await fs.rm(part, { force: true });
150+ await fs.rm(timingFile(videoId, takeId), { force: true });
151+ return true;
152+}
153+
154+// Do not leave ffmpeg capturing after the server exits.
155+process.on('exit', () => {
156+ for (const cap of captures.values()) cap.proc.kill('SIGKILL');
157+});
158+for (const sig of ['SIGINT', 'SIGTERM'] as const) {
159+ process.once(sig, () => {
160+ for (const cap of captures.values()) cap.proc.kill('SIGKILL');
161+ process.exit(0);
162+ });
163+}
server/takes.tsmodified+26−2View file
@@ -6,11 +6,12 @@ import type { Take, TrackKind } from '../shared/types.ts';
66 import { randomId } from '../shared/timeline.ts';
77 import { computePeaks, extractFrame, probe, remux } from './media.ts';
88 import { HttpError, exists, getConfig, getVideo, takeDir, updateVideo, videoDir } from './store.ts';
9+import { finalizeSystemAudio, startSystemCapture, stopSystemCapture, systemCaptureEvent } from './sysaudio.ts';
910 import { transcribeTake } from './transcribe.ts';
1011
1112 const TRACKS: TrackKind[] = ['screen', 'camera'];
1213
13-export async function createTake(videoId: string, input: { tracks: { screen: boolean; camera: boolean }; audio: TrackKind | null }): Promise<Take> {
14+export async function createTake(videoId: string, input: { tracks: { screen: boolean; camera: boolean }; audio: TrackKind | null; systemAudio?: boolean }): Promise<Take> {
1415 const tracks = { screen: !!input.tracks?.screen, camera: !!input.tracks?.camera };
1516 if (!tracks.screen && !tracks.camera) throw new HttpError(400, 'A take needs a screen or camera track');
1617 const audio = input.audio && tracks[input.audio] ? input.audio : null;
@@ -21,9 +22,18 @@ export async function createTake(videoId: string, input: { tracks: { screen: boo
2122 status: 'recording',
2223 tracks,
2324 audio,
25+ ...(input.systemAudio === true ? { systemAudio: true } : {}),
2426 duration: 0,
2527 };
2628 await fs.mkdir(takeDir(videoId, take.id), { recursive: true });
29+ if (take.systemAudio) {
30+ try {
31+ await startSystemCapture(videoId, take.id);
32+ } catch (e) {
33+ await fs.rm(takeDir(videoId, take.id), { recursive: true, force: true });
34+ throw new HttpError(500, (e as Error).message);
35+ }
36+ }
2737 await updateVideo(videoId, (v) => {
2838 v.takes.push(take);
2939 });
@@ -40,6 +50,7 @@ export async function appendChunk(videoId: string, takeId: string, track: string
4050 if (take.status !== 'recording') throw new HttpError(409, 'Take is not recording');
4151 if (!take.tracks[track]) throw new HttpError(400, `Take has no ${track} track`);
4252 await fs.appendFile(partFile(videoId, takeId, track), data);
53+ if (take.systemAudio) await systemCaptureEvent(videoId, takeId, null);
4354 }
4455
4556 const finalizing = new Set<string>();
@@ -75,13 +86,25 @@ export async function finalizeTake(videoId: string, takeId: string) {
7586 const duration = Math.min(...durations);
7687 if (!(duration > 0)) throw new Error('Recording is empty');
7788
78- const audioFile = take.audio ? path.join(dir, `${take.audio}.webm`) : null;
89+ // Computer audio is optional: if it cannot be aligned, keep the take without it.
90+ let systemAudio = false;
91+ if (take.systemAudio) {
92+ systemAudio = await finalizeSystemAudio(videoId, takeId, duration).catch((e) => {
93+ console.warn('system audio:', e.message);
94+ return false;
95+ });
96+ }
97+
98+ // The waveform shows the microphone, or the computer audio when there is no microphone.
99+ const audioFile = take.audio ? path.join(dir, `${take.audio}.webm`) : systemAudio ? path.join(dir, 'system.webm') : null;
79100 if (audioFile) await computePeaks(audioFile, path.join(dir, 'peaks.json')).catch((e) => console.warn('peaks:', e.message));
80101
81102 const updated = await updateVideo(videoId, (v) => {
82103 const t = v.takes.find((t) => t.id === takeId)!;
83104 t.status = 'ready';
84105 t.duration = duration;
106+ if (systemAudio) t.systemAudio = true;
107+ else delete t.systemAudio;
85108 delete t.error;
86109 v.timeline.push({ id: randomId(), takeId, start: 0, end: duration });
87110 });
@@ -137,6 +160,7 @@ export async function setThumbnail(videoId: string, jpeg: Buffer) {
137160 }
138161
139162 export async function deleteTake(videoId: string, takeId: string) {
163+ await stopSystemCapture(videoId, takeId);
140164 const updated = await updateVideo(videoId, (v) => {
141165 if (!v.takes.some((t) => t.id === takeId)) throw new HttpError(404, 'Take not found');
142166 v.takes = v.takes.filter((t) => t.id !== takeId);
shared/types.tsmodified+8−1View file
@@ -9,6 +9,8 @@ export interface Take {
99 tracks: { screen: boolean; camera: boolean };
1010 // Which track file carries the microphone audio (null if no mic was recorded).
1111 audio: TrackKind | null;
12+ // Computer audio (what the speakers played), in system.webm, aligned with the other tracks.
13+ systemAudio?: boolean;
1214 // Seconds; 0 until the take is finalized.
1315 duration: number;
1416 transcript?: { status: 'pending' | 'done' | 'failed'; error?: string; updatedAt: string };
@@ -69,8 +71,11 @@ export const isFullFrame = (c: Crop) => c.x < 0.001 && c.y < 0.001 && c.w > 0.99
6971 export interface OutputSettings {
7072 height: 720 | 1080;
7173 normalizeAudio: boolean;
74+ systemVolume?: number; // gain applied to computer audio in the mix, 0..1 (default 1)
7275 }
7376
77+export const systemVolumeOf = (o: OutputSettings) => o.systemVolume ?? 1;
78+
7479 export interface ExportInfo {
7580 file: string;
7681 exportedAt: string;
@@ -152,7 +157,7 @@ export interface Peaks {
152157 data: string;
153158 }
154159
155-export function exportSignature(v: Pick<Video, 'timeline' | 'layout' | 'output' | 'crops'>): string {
160+export function exportSignature(v: Pick<Video, 'timeline' | 'layout' | 'output' | 'crops' | 'takes'>): string {
156161 const seg = v.timeline.map((s) => [s.takeId, round3(s.start), round3(s.end)]);
157162 const l = v.layout;
158163 const o = v.output;
@@ -171,6 +176,8 @@ export function exportSignature(v: Pick<Video, 'timeline' | 'layout' | 'output'
171176 layout: [l.mode, l.corner, round3(l.size), l.shape],
172177 ...(b.mode === 'none' ? {} : { bg }),
173178 ...(Object.keys(crop).length ? { crop } : {}),
179+ // And the computer audio level only matters when a take in the timeline has computer audio.
180+ ...(v.timeline.some((s) => v.takes.find((t) => t.id === s.takeId)?.systemAudio) ? { sys: round3(systemVolumeOf(o)) } : {}),
174181 output: [o.height, o.normalizeAudio],
175182 });
176183 }
src/api.tsmodified+3−2View file
@@ -42,7 +42,8 @@ export const api = {
4242 patchVideo: (id: string, patch: Partial<Video>) => request<Video>('PATCH', v(id), patch),
4343 deleteVideo: (id: string) => request<{ ok: true }>('DELETE', v(id)),
4444
45- createTake: (id: string, input: { tracks: { screen: boolean; camera: boolean }; audio: TrackKind | null }) => request<Take>('POST', `${v(id)}/takes`, input),
45+ createTake: (id: string, input: { tracks: { screen: boolean; camera: boolean }; audio: TrackKind | null; systemAudio: boolean }) => request<Take>('POST', `${v(id)}/takes`, input),
46+ takeEvent: (id: string, takeId: string, ev: { type: 'start' | 'pause' | 'resume'; at: number } | null) => request<{ ok: true }>('POST', `${v(id)}/takes/${takeId}/events`, ev ?? {}),
4647 uploadChunk: (id: string, takeId: string, track: TrackKind, blob: Blob) => request<{ ok: true }>('POST', `${v(id)}/takes/${takeId}/chunk?track=${track}`, blob),
4748 finalizeTake: (id: string, takeId: string) => request<Video>('POST', `${v(id)}/takes/${takeId}/finalize`),
4849 deleteTake: (id: string, takeId: string) => request<Video>('DELETE', `${v(id)}/takes/${takeId}`),
@@ -57,5 +58,5 @@ export const api = {
5758 openFolder: (id: string) => request<{ ok: true; dir: string }>('POST', `${v(id)}/open-folder`),
5859 };
5960
60-export const takeFileUrl = (videoId: string, takeId: string, track: TrackKind | 'cutout', version?: string) =>
61+export const takeFileUrl = (videoId: string, takeId: string, track: TrackKind | 'cutout' | 'system', version?: string) =>
6162 `/media/${videoId}/takes/${takeId}/${track}.webm${version ? `?v=${encodeURIComponent(version)}` : ''}`;
src/components/Icon.tsxmodified+1−0View file
@@ -30,6 +30,7 @@ const PATHS: Record<string, string> = {
3030 trimStart: 'M5 4v16M9 12h10M13 8l-4 4 4 4',
3131 trimEnd: 'M19 4v16M15 12H5M11 8l4 4-4 4',
3232 crop: 'M6 2v14a2 2 0 0 0 2 2h14M18 22V8a2 2 0 0 0-2-2H2',
33+ speaker: 'M11 5L6 9H3v6h3l5 4zM15.5 8.5a5 5 0 0 1 0 7M18.5 5.5a9 9 0 0 1 0 13',
3334 };
3435
3536 export function Icon({ name, size = 16, className }: { name: keyof typeof PATHS | string; size?: number; className?: string }) {
src/components/Stage.tsxmodified+29−2View file
@@ -1,8 +1,8 @@
11 // Composite preview of the timeline, laid out in output pixels and scaled to fit, so it matches the export.
22
3-import { useEffect, useRef, useState, type CSSProperties, type SyntheticEvent } from 'react';
3+import { useCallback, useEffect, useRef, useState, type CSSProperties, type SyntheticEvent } from 'react';
44 import type { Crop, Take, Video } from '../../shared/types.ts';
5-import { cropOf } from '../../shared/types.ts';
5+import { cropOf, systemVolumeOf } from '../../shared/types.ts';
66 import { computeGeometry, outputSize, placeCropped, type Box, type Camera } from '../../shared/layout.ts';
77 import { takeFileUrl } from '../api.ts';
88 import { usePlayerActiveTake, type TimelinePlayer } from '../lib/player.ts';
@@ -61,6 +61,7 @@ export function Stage({ video, player }: { video: Video; player: TimelinePlayer
6161 <div key={t.id} style={{ display: t.id === active ? 'block' : 'none' }}>
6262 {t.tracks.screen && <ScreenLayer videoId={video.id} take={t} box={g.screen} crop={cropOf(video, t.id, 'screen')} player={player} />}
6363 {t.tracks.camera && <CameraLayer videoId={video.id} take={t} cam={g.camera} crop={cropOf(video, t.id, 'camera')} player={player} />}
64+ {t.systemAudio && <SystemAudio videoId={video.id} take={t} volume={systemVolumeOf(video.output)} player={player} />}
6465 </div>
6566 );
6667 })}
@@ -70,6 +71,32 @@ export function Stage({ video, player }: { video: Video; player: TimelinePlayer
7071 );
7172 }
7273
74+// The computer audio, played in sync with the take (a <video> element, since the player manages those).
75+function SystemAudio({ videoId, take, volume, player }: { videoId: string; take: Take; volume: number; player: TimelinePlayer }) {
76+ const ref = useRef<HTMLVideoElement | null>(null);
77+ useEffect(() => {
78+ if (ref.current) ref.current.volume = volume;
79+ }, [volume]);
80+ const register = player.refFor(take.id, 'system');
81+ // Stable, so that the player does not see the element re-registered (and re-seeked) on every render.
82+ const setRef = useCallback(
83+ (el: HTMLVideoElement | null) => {
84+ ref.current = el;
85+ register(el);
86+ },
87+ [register],
88+ );
89+ return (
90+ <video
91+ ref={setRef}
92+ src={takeFileUrl(videoId, take.id, 'system')}
93+ preload="auto"
94+ playsInline
95+ style={{ display: 'none' }}
96+ />
97+ );
98+}
99+
73100 // The screen track fit (letterboxed) into its box. The element stays mounted even when hidden, since it may carry the audio.
74101 function ScreenLayer({ videoId, take, box, crop, player }: { videoId: string; take: Take; box: Box | null; crop: Crop | undefined; player: TimelinePlayer }) {
75102 const [dims, onDims] = useVideoDims();
src/lib/player.tsmodified+3−3View file
@@ -11,8 +11,8 @@ export interface PlayerSnapshot {
1111 activeTakeId: string | null;
1212 }
1313
14-// Each take has up to three elements: the screen, the camera, and the camera with its background removed.
15-export type ElKey = TrackKind | 'cutout';
14+// Each take has up to four elements: the screen, the camera, the camera with its background removed, and the computer audio.
15+export type ElKey = TrackKind | 'cutout' | 'system';
1616 type Els = Partial<Record<ElKey, HTMLVideoElement>>;
1717
1818 export class TimelinePlayer {
@@ -79,7 +79,7 @@ export class TimelinePlayer {
7979
8080 private list(takeId: string): HTMLVideoElement[] {
8181 const e = this.els.get(takeId);
82- return e ? ([e.screen, e.camera, e.cutout].filter(Boolean) as HTMLVideoElement[]) : [];
82+ return e ? ([e.screen, e.camera, e.cutout, e.system].filter(Boolean) as HTMLVideoElement[]) : [];
8383 }
8484
8585 private master(takeId: string): HTMLVideoElement | undefined {
src/lib/recorder.tsmodified+24−1View file
@@ -57,8 +57,17 @@ interface Part {
5757
5858 export class RecordingSession {
5959 private parts: Part[] = [];
60+ private videoId: string;
61+ private takeId: string;
62+ private systemAudio: boolean;
63+ private keepAlive = 0;
6064
61- constructor(videoId: string, takeId: string, streams: Partial<Record<TrackKind, MediaStream>>, cameraHeight: number) {
65+ // With systemAudio, the server is capturing computer audio: it is told the wall-clock time of each start,
66+ // pause, and resume so that it can align that capture, and it gets a keep-alive while recording or paused.
67+ constructor(videoId: string, takeId: string, streams: Partial<Record<TrackKind, MediaStream>>, cameraHeight: number, systemAudio = false) {
68+ this.videoId = videoId;
69+ this.takeId = takeId;
70+ this.systemAudio = systemAudio;
6271 const mimeType = pickMimeType();
6372 for (const track of ['screen', 'camera'] as TrackKind[]) {
6473 const stream = streams[track];
@@ -77,20 +86,34 @@ export class RecordingSession {
7786 }
7887 }
7988
89+ private event(type: 'start' | 'pause' | 'resume') {
90+ if (!this.systemAudio) return;
91+ const ev = { type, at: Date.now() };
92+ // Retried like chunks, since a lost event would misalign the computer audio.
93+ const send = (attempt: number): Promise<unknown> =>
94+ api.takeEvent(this.videoId, this.takeId, ev).catch(() => (attempt < 8 ? sleep(300 * 2 ** attempt).then(() => send(attempt + 1)) : undefined));
95+ send(0);
96+ }
97+
8098 start() {
8199 for (const p of this.parts) p.recorder.start(1000);
100+ this.event('start');
101+ if (this.systemAudio) this.keepAlive = window.setInterval(() => api.takeEvent(this.videoId, this.takeId, null).catch(() => {}), 5000);
82102 }
83103
84104 pause() {
85105 for (const p of this.parts) if (p.recorder.state === 'recording') p.recorder.pause();
106+ this.event('pause');
86107 }
87108
88109 resume() {
89110 for (const p of this.parts) if (p.recorder.state === 'paused') p.recorder.resume();
111+ this.event('resume');
90112 }
91113
92114 // Stops recording and waits until every chunk has been uploaded.
93115 async stop() {
116+ clearInterval(this.keepAlive);
94117 for (const p of this.parts) if (p.recorder.state !== 'inactive') p.recorder.stop();
95118 await Promise.all(this.parts.map((p) => p.stopped));
96119 await Promise.all(this.parts.map((p) => p.uploader.flush()));
src/pages/Recorder.tsxmodified+13−1View file
@@ -16,6 +16,7 @@ interface Prefs {
1616 micId: string;
1717 audioProcessing: boolean;
1818 useScreen: boolean;
19+ useSystemAudio: boolean;
1920 countdown: number;
2021 }
2122
@@ -77,6 +78,7 @@ export function Recorder({ videoId }: { videoId: string }) {
7778 micId: '',
7879 audioProcessing: true,
7980 useScreen: true,
81+ useSystemAudio: false,
8082 countdown: cfg?.defaultCountdown ?? 3,
8183 }),
8284 );
@@ -172,6 +174,7 @@ export function Recorder({ videoId }: { videoId: string }) {
172174 const useCamera = !!prefs?.useCamera;
173175 const useScreen = !!prefs?.useScreen;
174176 const useMic = !!prefs?.useMic;
177+ const useSystemAudio = !!prefs?.useSystemAudio;
175178 const devicesReady = (useCamera || useScreen) && (!useCamera || !!camStream) && (!useMic || !!micStream);
176179 const canStart = phase === 'setup' && devicesReady;
177180
@@ -190,6 +193,7 @@ export function Recorder({ videoId }: { videoId: string }) {
190193 const take = await api.createTake(videoId, {
191194 tracks: { screen: useScreen, camera: useCamera },
192195 audio: useMic ? (useCamera ? 'camera' : 'screen') : null,
196+ systemAudio: useSystemAudio,
193197 });
194198 takeId.current = take.id;
195199 for (let n = prefs.countdown; n > 0; n--) {
@@ -206,7 +210,7 @@ export function Recorder({ videoId }: { videoId: string }) {
206210 const streams: Partial<Record<TrackKind, MediaStream>> = {};
207211 if (useCamera && camStream) streams.camera = new MediaStream([...camStream.getVideoTracks(), ...micTracks]);
208212 if (useScreen && screen) streams.screen = new MediaStream([...screen.getVideoTracks(), ...(useCamera ? [] : micTracks)]);
209- session.current = new RecordingSession(videoId, take.id, streams, prefs.cameraHeight);
213+ session.current = new RecordingSession(videoId, take.id, streams, prefs.cameraHeight, useSystemAudio);
210214 session.current.start();
211215 clock.current = { start: performance.now(), pausedAt: 0, pausedTotal: 0 };
212216 setElapsed(0);
@@ -403,6 +407,14 @@ export function Recorder({ videoId }: { videoId: string }) {
403407 {useMic && <MicMeter stream={micStream} />}
404408 </div>
405409
410+ <div className="source">
411+ <label className="check" title="What your computer plays through its speakers or headphones, recorded alongside the microphone">
412+ <input type="checkbox" checked={useSystemAudio} disabled={locked} onChange={(e) => setPrefs({ useSystemAudio: e.target.checked })} />
413+ <Icon name="speaker" /> Computer audio
414+ </label>
415+ {useSystemAudio && useMic && <p className="muted small">Use headphones, or the microphone will also pick up the sound from the speakers.</p>}
416+ </div>
417+
406418 <div className="source">
407419 <label className="check">
408420 <input type="checkbox" checked={useScreen} disabled={locked} onChange={(e) => setPrefs({ useScreen: e.target.checked })} />
src/pages/VideoPage.tsxmodified+8−1View file
@@ -1,6 +1,6 @@
11 import { useCallback, useEffect, useMemo, useRef, useState } from 'react';
22 import type { Background, ExportJob, Layout, OutputSettings, PublicConfig, Segment, Take, TakeCrop, Transcript, Video } from '../../shared/types.ts';
3-import { backgroundOf, exportSignature } from '../../shared/types.ts';
3+import { backgroundOf, exportSignature, systemVolumeOf } from '../../shared/types.ts';
44 import { computeGeometry } from '../../shared/layout.ts';
55 import { cutRange, locate, moveSegment, randomId, segDuration, segmentStarts, splitAt, timelineDuration } from '../../shared/timeline.ts';
66 import { navigate } from '../App.tsx';
@@ -664,6 +664,12 @@ export function VideoPage({ videoId }: { videoId: string }) {
664664 <input type="checkbox" checked={video.output.normalizeAudio} onChange={(e) => setOutput({ normalizeAudio: e.target.checked })} />
665665 Normalize loudness (−16 LUFS)
666666 </label>
667+ {readyTakes.some((t) => t.systemAudio) && (
668+ <label className="field">
669+ <span>Computer audio level ({Math.round(systemVolumeOf(video.output) * 100)}%)</span>
670+ <input type="range" min={0} max={1} step={0.05} value={systemVolumeOf(video.output)} onChange={(e) => setOutput({ systemVolume: parseFloat(e.target.value) })} />
671+ </label>
672+ )}
667673
668674 <div className="export-box">
669675 {job?.status === 'running' ? (
@@ -795,6 +801,7 @@ function TakeCard({
795801 {take.tracks.camera && <Icon name="camera" size={14} />}
796802 {take.tracks.screen && <Icon name="screen" size={14} />}
797803 {take.audio && <Icon name="mic" size={14} />}
804+ {take.systemAudio && <Icon name="speaker" size={14} />}
798805 </span>
799806 </div>
800807 <div className="small muted">