/ proof-of-concept / takes
Sign in
proof-of-concept / takes
6ec86a8
Branches
main
takes / README.md
12.8 KBPreviewCodeBlameHistoryRaw

Takes#

Takes is a small local application for recording, organizing, and editing short explanatory videos on Linux. A video can contain camera footage of the speaker, a screen recording, or both at once. Each video has a title, a description, and notes, and it is built from one or more recorded takes that are trimmed, cut, and joined on a timeline before being rendered to MP4.

We wrote it for a specific workflow: explaining or demonstrating something in a few minutes, possibly over several attempts, with as little friction as possible between having an idea and having a shareable file. Unlike general-purpose tools such as OBS (recording) combined with Kdenlive (editing), Takes keeps the library, the recorder, and a deliberately limited editor in one place. The trade-off is that it does much less: there are no scenes, transitions, titles, or multi-track audio.

Requirements#

Running#

npm install
npm run build
npm start            # or: node server/index.ts

To install a takes command:

npm link
takes                # opens a Chrome app window at http://localhost:4455

Options: --dir DIR (library directory, default ~/Videos/takes), --port PORT (default 4455), --no-open, and --browser (use the default browser instead of a Chrome app window). If Takes is already running, takes only opens a new window.

For development, npm run dev runs the server on port 4455 with automatic restarts and the Vite dev server on http://localhost:5173.

Using it#

The library shows every video with its thumbnail, duration, and description. Drag cards to reorder them. Deleting a video (with confirmation) moves its folder to .trash/ inside the library, so a mistaken deletion can be undone by hand.

Recording. Choose any combination of camera, microphone, and screen, then start recording. The screen picker is GNOME's own portal dialog, which lets you choose a monitor or a single window. A countdown precedes the recording, and it can be paused and resumed. The Teleprompter window button opens the notes in a separate window with large text, optional auto-scroll, and start/pause/stop controls, so it can sit on a second monitor or outside the captured window. Recorded data is uploaded to the server once per second while recording, so a browser crash loses at most about a second; an interrupted take appears on the video page with a Recover button.

Microphone level. Below the microphone is a level meter in dBFS, with a shaded band from −12 to −6 dB, the target range for the peaks of speech, a line that holds the recent peak, and a Clipping warning. With PipeWire, a Gain slider sets the microphone's input gain in the system (through wpctl, on the node whose description matches the browser's label for the microphone; PipeWire moves the sound card's capture control along with it). Check level measures the level for 6 seconds while you speak, reports the speech peaks (the 95th percentile of the peak level over 43 ms blocks) and the background noise (the 10th percentile of the RMS level), and suggests a gain change toward −9 dB, which can be applied with one click. The suggestion converts decibels to wpctl's volume scale, which is only approximately right for hardware controls, so check again after applying it. Export normalizes the loudness anyway, so the aim is mainly to avoid clipping, which cannot be undone, and a level so low that normalization brings up the background noise. Automatic gain (off by default) lets the browser adjust the level instead; it is separate from Noise suppression, since it would otherwise move the gain that was set by hand.

Computer audio. With Computer audio checked, Takes also records what the computer plays through its default output (for example, the sound of a demo), in addition to the microphone. Unlike the microphone, this is not captured by the browser: the server records the PipeWire or PulseAudio monitor of the default output with ffmpeg, so it works in any browser and without sharing the screen. The two recordings are aligned by wall-clock time, which the browser and the server share since they run on the same machine. ffmpeg reports when its first sample was captured, the recorder reports when it started, paused, and resumed, and when the take is saved the capture is cut to the same intervals and stored as system.webm. In a test with a synthetic capture, the cut points were within 15 ms of where they should be. Computer audio is kept separate from the microphone: it is mixed in during playback and export, with its own level on the Layout & export tab, and it is not transcribed. Use headphones when recording both, since otherwise the microphone also picks up the sound from the speakers.

Editing. Each video has a timeline made of segments of its takes. A new take is appended to the end of the timeline, and a take can be added again from the Takes list. The editor supports splitting at the playhead (S), marking a range (I and O) and cutting it (X), trimming the current segment to the playhead, removing and reordering segments, and undo/redo. The timeline shows the audio waveform, and a zoom slider helps with precise cuts. Editing is non-destructive: the raw takes are never modified.

Transcripts. With a Deepgram API key (set it on the Settings page), each take's audio is transcribed after recording. The Transcript tab shows the words that remain in the timeline. Clicking a word seeks to it, and shift-clicking a second word selects the range between them, which can then be cut. This is often the quickest way to remove a false start or a filler word.

Layout and export. For takes that have both camera and screen, the camera can be placed as picture-in-picture (corner, size, and rectangle, rounded, or circle shape), side by side, or hidden. The preview in the editor uses the same geometry as the export. Export renders the timeline to an H.264/AAC MP4 at 720p or 1080p, optionally with loudness normalization to −16 LUFS, and marks the export as outdated when the timeline or layout changes afterwards.

Cropping. The Crop button on a take opens a frame of its screen or camera track with a rectangle that can be moved, resized, or redrawn by dragging; a slider picks the frame to show. This is useful for keeping only one window of a full-screen recording, or for reframing the camera. The rectangle can be free or locked to an aspect ratio, including Fill layout, the shape of the area that track occupies in the current layout, so that the cropped screen is not letterboxed. The crop is fit into the layout exactly as the full frame would be, and Apply to all takes copies it to the other takes. Like the timeline, cropping is non-destructive: it is stored as fractions of the frame in video.json, so it does not depend on the recorded resolution, and background removal does not need to run again when it changes.

Background removal. In Layout & export, the background behind you can be kept, blurred, replaced with a solid color, or removed entirely (Cutout, for picture-in-picture, where only you appear over the screen recording, sitting on the bottom edge of the video). Choosing any of these processes the camera track of each take with Robust Video Matting (Lin et al., 2021), a recurrent segmentation model designed for video, which gives soft edges around hair and little flicker between frames. The model runs on the CPU through onnxruntime; takes are processed one at a time in the background, and the result is stored as cutout.webm (VP9 with an alpha channel) so that each take is processed only once. The model (about 15 MB) is downloaded from GitHub to ~/.cache/takes/models/ the first time it is needed. Export waits until every take in the timeline has been processed. A running or queued job can be cancelled; a cancelled take is not processed again until you start it.

Processing resolution. The camera is scaled down before background removal to at most the number of lines chosen under Processing resolution (240 to 1080, default 480), and the model's coarse pass shrinks with it, since that pass takes most of the time. A small picture-in-picture needs far fewer lines than the camera records: at 720p output, a camera box of a quarter of the width is 320×180 px. On the development laptop, a 426×240 frame took 76 ms, compared with 250 ms for 1280×720. We have not measured how much the matte quality drops at the lowest settings. Changing the setting restarts a running job; takes already processed at a lower resolution offer a Redo button, and takes processed at a higher one are kept.

Background removal on another machine. On a laptop CPU, background removal is slow, so it can instead run on a machine with an NVIDIA GPU that you reach over ssh. Set ssh host on the Settings page to an ssh destination such as office or user@host; it must accept key-based login without prompting, since Takes runs ssh without a terminal. That machine needs a working NVIDIA driver, Python 3.10 or later, and ffmpeg (with libvpx) on the PATH. If these come from environment modules or similar, put the commands that load them in Setup command (for example module load python ffmpeg); they run in a bash login shell before each job. For each take, Takes sends the camera track over a single ssh connection, first scaling it down to the processing resolution (see below) and re-encoding it as H.264 at CRF 18 when that is smaller than the camera; at 240 lines this cut the upload of a 720p camera track by a factor of about 15. It then runs the same model with onnxruntime-gpu in a temporary directory there, receives cutout.webm, and removes the temporary directory. The first job creates a Python environment in ~/.cache/takes/venv on the remote machine (about 2.5 GB, because pip installs the CUDA and cuDNN libraries); delete that directory to rebuild it. For a 60 s, 640×480 take, processing took 27.7 s on a remote RTX 6000 Ada, including the transfer, compared with 536 s on the CPU of a 12-thread laptop; the two cutouts differ by less than 1/255 in alpha on average. (That comparison was made at full resolution, without the re-encoding step, which adds a small difference of its own.)

Storage layout#

~/Videos/takes/
  library.json                       order of videos
  config.json                        settings, including the Deepgram key (mode 0600)
  <video-id>/
    video.json                       title, description, notes, takes, timeline, layout, crops
    thumbnail.jpg
    <title>.mp4                      latest export
    takes/<take-id>/
      screen.webm  camera.webm       raw recordings (mic audio is in camera.webm when there is a camera)
      system.webm                    computer audio, aligned with the recordings, if requested
      cutout.webm                    camera with background removed (VP9 + alpha), if requested
      peaks.json                     waveform data
      transcript.json                Deepgram words with timestamps

Everything is plain files, so videos can be backed up, synced, or inspected without the application.

Limitations#

↑↓ move↵openescclose