Export at 1440p and 2160p as well as 720p and 1080p; the output settings say which height shows the screen recording at full size (screen size stored per take, probed once for older takes without touching updatedAt); the screen recording bitrate grows with the pixel count, 6 to 20 Mbit/s - proof-of-concept/takes

proof-of-concept / takes

Export at 1440p and 2160p as well as 720p and 1080p; the output settings say which height shows the screen recording at full size (screen size stored per take, probed once for older takes without touching updatedAt); the screen recording bitrate grows with the pixel count, 6 to 20 Mbit/s

Jeremy Magland <jmagland@flatironinstitute.org> committed 13 days ago commit 6a456f2d7996 parent 969e06c Browse files

8 changed files+91−17

README.mdmodified+1−1View file

@@ -44,7 +44,7 @@ The library shows every video with its thumbnail, duration, and description. Dra

4444

4545 **Transcripts.** With a Deepgram API key (set it on the Settings page), each take's audio is transcribed after recording. The *Transcript* tab shows the words that remain in the timeline. Clicking a word seeks to it, and shift-clicking a second word selects the range between them, which can then be cut. This is often the quickest way to remove a false start or a filler word.

4646

47-**Layout and export.** For takes that have both camera and screen, the camera can be placed as picture-in-picture (corner, size, and rectangle, rounded, or circle shape), side by side, or hidden. The preview in the editor uses the same geometry as the export. Export renders the timeline to an H.264/AAC MP4 at 720p or 1080p, optionally with loudness normalization to −16 LUFS, and marks the export as outdated when the timeline or layout changes afterwards.

47+**Layout and export.** For takes that have both camera and screen, the camera can be placed as picture-in-picture (corner, size, and rectangle, rounded, or circle shape), side by side, or hidden. The preview in the editor uses the same geometry as the export. Export renders the timeline to an H.264/AAC MP4 at 720p, 1080p, 1440p, or 2160p, optionally with loudness normalization to −16 LUFS, and marks the export as outdated when the timeline or layout changes afterwards. The screen is recorded at its native size, with a bitrate that grows with the pixel count (6 Mbit/s for 1920×1080, up to 20 Mbit/s for 4K), and the output settings say at which resolution the export shows the screen at full size rather than scaling it down. A 16:10 screen is letterboxed in the 16:9 output, so a 2560×1600 screen needs 2160p to be shown at full size.

4848

4949 **Cropping.** The *Crop* button on a take opens a frame of its screen or camera track with a rectangle that can be moved, resized, or redrawn by dragging; a slider picks the frame to show. This is useful for keeping only one window of a full-screen recording, or for reframing the camera. The rectangle can be free or locked to an aspect ratio, including *Fill layout*, the shape of the area that track occupies in the current layout, so that the cropped screen is not letterboxed. The crop is fit into the layout exactly as the full frame would be, and *Apply to all takes* copies it to the other takes. Like the timeline, cropping is non-destructive: it is stored as fractions of the frame in `video.json`, so it does not depend on the recorded resolution, and background removal does not need to run again when it changes.

5050

server/index.tsmodified+5−4View file

@@ -10,10 +10,10 @@ import os from 'node:os';

1010 import path from 'node:path';

1111 import { fileURLToPath } from 'node:url';

1212 import type { Crop, PublicConfig, TakeCrop, Video } from '../shared/types.ts';

13-import { MATTE_RESOLUTIONS, backgroundOf, isFullFrame } from '../shared/types.ts';

13+import { MATTE_RESOLUTIONS, backgroundOf, isFullFrame, isOutputHeight } from '../shared/types.ts';

1414 import { getExportJob, startExport } from './export.ts';

1515 import { HttpError, checkId, createVideo, getConfig, getVideo, initStore, libraryDir, listVideos, setOrder, trashVideo, updateConfig, updateVideo, videoDir } from './store.ts';

16-import { appendChunk, createTake, deleteTake, finalizeTake, setThumbnail } from './takes.ts';

16+import { appendChunk, backfillScreenSizes, createTake, deleteTake, finalizeTake, setThumbnail } from './takes.ts';

1717 import { systemCaptureEvent } from './sysaudio.ts';

1818 import { transcribeTake } from './transcribe.ts';

1919 import { cancelMattes, requestMattes, withMatteStatus } from './matte.ts';

@@ -82,7 +82,7 @@ app.put('/api/config', json, async (req, res) => {

8282 if (typeof b.autoTranscribe === 'boolean') patch.autoTranscribe = b.autoTranscribe;

8383 if (typeof b.defaultCountdown === 'number') patch.defaultCountdown = Math.max(0, Math.min(10, Math.round(b.defaultCountdown)));

8484 if (b.defaultCameraHeight === 720 || b.defaultCameraHeight === 1080) patch.defaultCameraHeight = b.defaultCameraHeight;

85- if (b.defaultOutputHeight === 720 || b.defaultOutputHeight === 1080) patch.defaultOutputHeight = b.defaultOutputHeight;

85+ if (isOutputHeight(b.defaultOutputHeight)) patch.defaultOutputHeight = b.defaultOutputHeight;

8686 if (typeof b.matteHost === 'string') patch.matteHost = b.matteHost.trim();

8787 if (typeof b.matteSetup === 'string') patch.matteSetup = b.matteSetup.trim();

8888 await updateConfig(patch);

@@ -114,7 +114,7 @@ app.put('/api/order', json, async (req, res) => {

114114 });

115115

116116 app.get('/api/videos/:id', async (req, res) => {

117- res.json(withMatteStatus(await getVideo(p(req, 'id'))));

117+ res.json(withMatteStatus(await backfillScreenSizes(p(req, 'id'))));

118118 });

119119

120120 // Editable fields only; takes are managed through their own endpoints.

@@ -139,6 +139,7 @@ app.patch('/api/videos/:id', json, async (req, res) => {

139139 }

140140 if (b.output) {

141141 v.output = { ...v.output, ...b.output };

142+ if (!isOutputHeight(v.output.height)) throw new HttpError(400, 'Invalid output resolution');

142143 const sv = v.output.systemVolume;

143144 if (sv !== undefined && !(typeof sv === 'number' && sv >= 0 && sv <= 1)) throw new HttpError(400, 'Invalid computer audio level');

144145 }

server/store.tsmodified+3−2View file

@@ -114,11 +114,12 @@ export async function getVideo(id: string): Promise<Video> {

114114 return v;

115115 }

116116

117-export function updateVideo(id: string, fn: (v: Video) => void | Promise<void>): Promise<Video> {

117+// `touch: false` is for bookkeeping that is not an edit (the library shows updatedAt).

118+export function updateVideo(id: string, fn: (v: Video) => void | Promise<void>, { touch = true } = {}): Promise<Video> {

118119 return withLock(id, async () => {

119120 const v = await getVideo(id);

120121 await fn(v);

121- v.updatedAt = new Date().toISOString();

122+ if (touch) v.updatedAt = new Date().toISOString();

122123 await writeJsonAtomic(path.join(videoDir(id), 'video.json'), v);

123124 return v;

124125 });

server/takes.tsmodified+26−2View file

@@ -2,7 +2,7 @@

22

33 import fs from 'node:fs/promises';

44 import path from 'node:path';

5-import type { Take, TrackKind } from '../shared/types.ts';

5+import type { Take, TrackKind, Video } from '../shared/types.ts';

66 import { randomId } from '../shared/timeline.ts';

77 import { computePeaks, extractFrame, probe, remux } from './media.ts';

88 import { HttpError, exists, getConfig, getVideo, takeDir, updateVideo, videoDir } from './store.ts';

@@ -71,6 +71,7 @@ export async function finalizeTake(videoId: string, takeId: string) {

7171

7272 const dir = takeDir(videoId, takeId);

7373 const durations: number[] = [];

74+ let screenSize: { width: number; height: number } | undefined;

7475 try {

7576 for (const track of TRACKS) {

7677 if (!take.tracks[track]) continue;

@@ -81,7 +82,9 @@ export async function finalizeTake(videoId: string, takeId: string) {

8182 await fs.rm(part);

8283 }

8384 if (!(await exists(final))) throw new Error(`No ${track} data was recorded`);

84- durations.push((await probe(final)).duration);

85+ const info = await probe(final);

86+ durations.push(info.duration);

87+ if (track === 'screen' && info.width && info.height) screenSize = { width: info.width, height: info.height };

8588 }

8689 const duration = Math.min(...durations);

8790 if (!(duration > 0)) throw new Error('Recording is empty');

@@ -103,6 +106,7 @@ export async function finalizeTake(videoId: string, takeId: string) {

103106 const t = v.takes.find((t) => t.id === takeId)!;

104107 t.status = 'ready';

105108 t.duration = duration;

109+ if (screenSize) t.screenSize = screenSize;

106110 if (systemAudio) t.systemAudio = true;

107111 else delete t.systemAudio;

108112 delete t.error;

@@ -171,3 +175,23 @@ export async function deleteTake(videoId: string, takeId: string) {

171175 if (updated.thumbnail && !updated.thumbnail.custom) await autoThumbnail(videoId);

172176 return getVideo(videoId);

173177 }

178+

179+// The screen size of takes recorded before it was stored, probed once from screen.webm and saved,

180+// so the output settings can say which resolution shows the screen at full size.

181+export async function backfillScreenSizes(videoId: string): Promise<Video> {

182+ const v = await getVideo(videoId);

183+ const missing = v.takes.filter((t) => t.status === 'ready' && t.tracks.screen && !t.screenSize);

184+ if (missing.length === 0) return v;

185+ const sizes = new Map<string, { width: number; height: number }>();

186+ for (const t of missing) {

187+ const info = await probe(path.join(takeDir(videoId, t.id), 'screen.webm')).catch(() => null);

188+ if (info?.width && info.height) sizes.set(t.id, { width: info.width, height: info.height });

189+ }

190+ if (sizes.size === 0) return v;

191+ return updateVideo(videoId, (v) => {

192+ for (const t of v.takes) {

193+ const size = sizes.get(t.id);

194+ if (size && !t.screenSize) t.screenSize = size;

195+ }

196+ }, { touch: false });

197+}

shared/types.tsmodified+10−2View file

@@ -21,6 +21,8 @@ export interface Take {

2121 // of lines the cutout was processed at, and `capped` says whether that was limited by the

2222 // processing resolution setting rather than by the camera (both absent for older cutouts).

2323 matte?: { status: 'queued' | 'running' | 'done' | 'failed' | 'cancelled'; stage?: string; progress?: number; error?: string; height?: number; capped?: boolean; updatedAt: string };

24+ // Pixel size of the screen track, as recorded (absent for takes from before it was stored).

25+ screenSize?: { width: number; height: number };

2426 error?: string;

2527 }

2628

@@ -82,8 +84,14 @@ export const cropOf = (v: Pick<Video, 'crops'>, takeId: string, track: TrackKind

8284

8385 export const isFullFrame = (c: Crop) => c.x < 0.001 && c.y < 0.001 && c.w > 0.999 && c.h > 0.999;

8486

87+// Export heights; the width is always 16:9. 1440p and 2160p show a high-resolution screen

88+// recording without scaling it down, at the cost of slower exports and larger files.

89+export const OUTPUT_HEIGHTS = [720, 1080, 1440, 2160] as const;

90+export type OutputHeight = (typeof OUTPUT_HEIGHTS)[number];

91+export const isOutputHeight = (h: unknown): h is OutputHeight => OUTPUT_HEIGHTS.includes(h as OutputHeight);

92+

8593 export interface OutputSettings {

86- height: 720 | 1080;

94+ height: OutputHeight;

8795 normalizeAudio: boolean;

8896 systemVolume?: number; // gain applied to computer audio in the mix, 0..1 (default 1)

8997 }

@@ -140,7 +148,7 @@ export interface AppConfig {

140148 autoTranscribe: boolean;

141149 defaultCountdown: number;

142150 defaultCameraHeight: 720 | 1080;

143- defaultOutputHeight: 720 | 1080;

151+ defaultOutputHeight: OutputHeight;

144152 matteHost: string; // ssh destination for background removal on a GPU; empty for this computer

145153 matteSetup: string; // shell commands run on that host first, e.g. "module load python ffmpeg"

146154 }

src/lib/recorder.tsmodified+10−1View file

@@ -74,7 +74,7 @@ export class RecordingSession {

7474 if (!stream) continue;

7575 const recorder = new MediaRecorder(stream, {

7676 mimeType,

77- videoBitsPerSecond: track === 'screen' ? 6_000_000 : cameraHeight >= 1080 ? 5_000_000 : 3_000_000,

77+ videoBitsPerSecond: track === 'screen' ? screenBitrate(stream) : cameraHeight >= 1080 ? 5_000_000 : 3_000_000,

7878 audioBitsPerSecond: 128_000,

7979 });

8080 const uploader = new ChunkUploader(videoId, takeId, track);

@@ -123,3 +123,12 @@ export class RecordingSession {

123123 return this.parts.find((p) => p.uploader.error)?.uploader.error ?? null;

124124 }

125125 }

126+

127+// 6 Mbit/s for a 1080p screen, raised in proportion to the pixel count for larger screens (a

128+// 2560x1600 laptop panel gets about 12, a 4K monitor 20), so that small text stays sharp when the

129+// export keeps the screen at full size.

130+function screenBitrate(stream: MediaStream): number {

131+ const { width, height } = stream.getVideoTracks()[0]?.getSettings() ?? {};

132+ const pixels = width && height ? width * height : 1920 * 1080;

133+ return Math.round(Math.min(20_000_000, Math.max(6_000_000, (6_000_000 * pixels) / (1920 * 1080))));

134+}

src/pages/Settings.tsxmodified+6−2View file

@@ -1,5 +1,6 @@

11 import { useEffect, useState } from 'react';

22 import type { PublicConfig } from '../../shared/types.ts';

3+import { OUTPUT_HEIGHTS } from '../../shared/types.ts';

34 import { api } from '../api.ts';

45 import { Icon } from '../components/Icon.tsx';

56

@@ -117,8 +118,11 @@ export function Settings() {

117118 <label className="row">

118119 <span>Export resolution for new videos</span>

119120 <select value={cfg.defaultOutputHeight} onChange={(e) => save({ defaultOutputHeight: parseInt(e.target.value) })}>

120- <option value={720}>720p</option>

121- <option value={1080}>1080p</option>

121+ {OUTPUT_HEIGHTS.map((h) => (

122+ <option key={h} value={h}>

123+ {h}p

124+ </option>

125+ ))}

122126 </select>

123127 </label>

124128 <p className="muted small">The recorder remembers your last choices; these defaults apply until you change them there.</p>

src/pages/VideoPage.tsxmodified+30−3View file

@@ -1,7 +1,7 @@

11 import { useCallback, useEffect, useMemo, useRef, useState } from 'react';

22 import type { Background, ExportJob, Layout, OutputSettings, PublicConfig, Segment, Take, TakeCrop, Transcript, Video } from '../../shared/types.ts';

3-import { MATTE_RESOLUTIONS, backgroundOf, cropOf, exportSignature, matteBelowSetting, systemVolumeOf } from '../../shared/types.ts';

4-import { computeGeometry } from '../../shared/layout.ts';

3+import { MATTE_RESOLUTIONS, OUTPUT_HEIGHTS, backgroundOf, cropOf, exportSignature, matteBelowSetting, systemVolumeOf } from '../../shared/types.ts';

4+import { computeGeometry, placeCropped } from '../../shared/layout.ts';

55 import { cutRange, locate, moveSegment, randomId, segDuration, segmentStarts, splitAt, timelineDuration } from '../../shared/timeline.ts';

66 import { navigate } from '../App.tsx';

77 import { api } from '../api.ts';

@@ -373,6 +373,23 @@ export function VideoPage({ videoId }: { videoId: string }) {

373373 const lines = Math.round(cam.box.h / (cropOf(video, t.id, 'camera')?.h ?? 1));

374374 return !best || lines > best.lines ? { w: cam.box.w, h: cam.box.h, lines } : best;

375375 }, null);

376+ // How much the export scales the screen down, for the timeline take where it is scaled most, at

377+ // each output height: 1 or more shows the recording at full size. Takes recorded before the

378+ // screen size was stored are left out until the server has probed them.

379+ const screenScale = (take: Take, height: OutputSettings['height']) => {

380+ const box = computeGeometry(video.layout, { ...video.output, height }, true, take.tracks.camera).screen;

381+ if (!box || !take.screenSize) return null;

382+ const { width, height: h } = take.screenSize;

383+ const crop = cropOf(video, take.id, 'screen');

384+ return placeCropped(box, width, h, crop, 'contain').clip.w / ((crop?.w ?? 1) * width);

385+ };

386+ const screenFit = [...new Set(video.timeline.map((s) => s.takeId))]

387+ .map((id) => readyTakes.find((t) => t.id === id))

388+ .reduce<{ take: Take; scale: number } | null>((worst, t) => {

389+ const scale = t ? screenScale(t, video.output.height) : null;

390+ return t && scale !== null && (!worst || scale < worst.scale) ? { take: t, scale } : worst;

391+ }, null);

392+ const fullSizeAt = screenFit ? OUTPUT_HEIGHTS.find((h) => (screenScale(screenFit.take, h) ?? 0) >= 0.995) : undefined;

376393 // Takes in the timeline whose camera appears in the output but has not had its background removed yet.

377394 const matteBlocking =

378395 background.mode === 'none'

@@ -696,13 +713,23 @@ export function VideoPage({ videoId }: { videoId: string }) {

696713 <div className="field">

697714 <span>Resolution</span>

698715 <div className="seg-control">

699- {([720, 1080] as const).map((h) => (

716+ {OUTPUT_HEIGHTS.map((h) => (

700717 <button key={h} className={video.output.height === h ? 'active' : ''} onClick={() => setOutput({ height: h })}>

701718 {h}p

702719 </button>

703720 ))}

704721 </div>

705722 </div>

723+ {screenFit?.take.screenSize && (

724+ <p className="muted small">

725+ The screen was recorded at {screenFit.take.screenSize.width}×{screenFit.take.screenSize.height}

726+ {screenFit.scale >= 0.995

727+ ? `, which ${video.output.height}p shows at full size.`

728+ : `; ${video.output.height}p shows it at ${Math.round(screenFit.scale * 100)}% of that size. ` +

729+ (fullSizeAt ? `${fullSizeAt}p shows it at full size.` : `Even ${OUTPUT_HEIGHTS[OUTPUT_HEIGHTS.length - 1]}p scales it down.`)}{' '}

730+ Higher resolutions export more slowly and make larger files.

731+ </p>

732+ )}

706733 <label className="check">

707734 <input type="checkbox" checked={video.output.normalizeAudio} onChange={(e) => setOutput({ normalizeAudio: e.target.checked })} />

708735 Normalize loudness (−16 LUFS)