Skip to content

MCP.md

Latest commit

 

History

History
652 lines (533 loc) · 58.7 KB

MCP.md

File metadata and controls

652 lines (533 loc) · 58.7 KB
Raw

Agent access (Drift MCP)

Drift can expose a localhost MCP server so Cursor, Claude Code, or other agents can edit the open project. Enable it in Settings → Agent access (off at every launch by default; a "Start agent on startup" switch there opts into starting it automatically instead, and resets itself the next time access is turned off).

Connect

Cursor / Claude (this session): copy the snippet from Settings after enabling. The bearer token rotates each session.

One-time stdio attach (no token in mcp.json):

{
  "mcpServers": {
    "drift": {
      "command": "/path/to/drift",
      "args": ["--mcp-stdio"]
    }
  }
}

drift --mcp-stdio attaches to a running editor with Agent access enabled. It speaks newline-delimited JSON-RPC, the framing the MCP stdio transport specifies. If the editor is closed or Agent access is off, each call comes back as a JSON-RPC error and the bridge keeps running, so enabling Agent access is enough to make it work — no need to restart the client.

Headless

drift --headless runs the editor with no window and serves MCP itself, for servers and unattended automation. It needs no running editor, no session token and no open project:

{
  "mcpServers": {
    "drift": {
      "command": "/path/to/drift",
      "args": ["--headless"]
    }
  }
}
Command Transport
drift --headless [project.json] MCP on stdin/stdout; quits when stdin closes
drift --headless --mcp-port 4731 [--mcp-token T] MCP over HTTP; runs until SIGINT/SIGTERM
drift --headless --mcp-stdio --mcp-port 4731 both

On startup it prints what is running — platform plugin, OpenGL, project, transports, and the HTTP URL, token and session-file path. That banner goes to stdout, except when stdio is a transport, where it goes to stderr instead: the spec allows nothing but MCP messages on stdout, and a banner there is exactly what breaks clients.

Without --mcp-token (or $DRIFT_MCP_TOKEN) a token is generated and shown in the banner. Only the HTTP form writes the session file that --mcp-stdio reads, so a headless instance serving stdio never disturbs a GUI editor running alongside it.

Both forms shut down on SIGINT (Ctrl-C) and SIGTERM.

Rendering needs a GL context, which is not the same as needing a window. The compositor wants OpenGL 3.3 core on an offscreen surface, and Qt's offscreen platform plugin cannot provide one on a host with no /dev/dri. On a headless Linux box run:

QT_QPA_PLATFORM=xcb xvfb-run -a drift --headless

Mesa's llvmpipe covers 3.3 in software; a GPU host can use an EGL platform plugin instead. Started without a usable context, Drift still serves MCP and still edits projects — it says so on stderr, and only render, capture and export fail.

Marketplace consent is granted in the GUI only. A headless instance answers every market op with consent_required unless the same settings profile already carries market/consented=true from a GUI session that accepted the terms — there is deliberately no op that accepts them.

Workflow

The initialize reply carries a short instructions string with the essentials; the eight homepage tools are:

  1. catalog — toolboxes and their ops as "name — when" strings, plus limitations (~3k tokens). catalog({brief:true}) is names only; catalog({guide:true}) adds the long prose guide; catalog({endpoints:true}) adds the HTTP endpoints.
  2. search({q}) — find ops by keyword across names, when-hints, descriptions and argument names. Returns hits:[{name, toolbox, when, args, required}]; schema:true inlines the full schema when there are ≤3 hits. Use this instead of loading a whole toolbox.
  3. toolbox({name}) or toolbox({ops:[…]}) — full JSON schemas for a toolbox, or for just the named ops (any toolbox).
  4. apply({ops:[{tool, args}, …]}) — run mutations in order; one undo step for the batch.
  5. inspect() — project summary; clips:true adds clip rows, detail:true expands them, clip:<uuid> / track:<n> filter, since:<revision> short-circuits.
  6. activity() — cheap text profile of visual change, motion and loudness over a range; tells you where to look.
  7. frames() — one labelled contact-sheet image of many moments; tells you what is there.
  8. capture({at}) — one full-size JPEG still of the composition (use to verify edits).

Homepage endpoint: POST /mcp with Authorization: Bearer <token>.

Pinned endpoints (/mcp/timeline, /mcp/project, …) list that toolbox’s ops directly. catalog, search, toolbox, and apply are only on /mcp; inspect, capture, frames and activity work on both. Toolbox ops can also be called by name directly on /mcp instead of through apply — but only apply collapses a batch into one undo step.

apply takes toolbox ops only. The eight homepage tools return unknown_op inside an ops array; call them directly. get_waveform({image:true}) is also refused inside apply.

Conventions

Topic Rule
Time Seconds
Clip reference Prefer clip UUID from inspect; else track (0 = top) + index. One or the other is required — clip ops never fall back to the selection. Read the current selection from inspect.selection
Selection ops separate_audio, unlink_audio, merge_clips, align_clip_left/right, copy_selection, cut_selection take no clip argument — call select_clip first. merge_clips joins two abutting cuts of the same media, or two or more subtitle clips on one track (select_clips; gaps allowed, the earliest clip's style and transform win); merge_clips({track}) merges every subtitle clip on that track without a selection. freeze_frame and paste_at_playhead are playhead-based (seek first); they do not use the selection
Discovery Effect stack indices, transition ids, and bookmark indices exist only in inspect({clips:true, detail:true}). Subtitle cues: inspect({clips:true, cues:true}) or the subtitleCues field of a detail row. Mask/fade/speed/volume/keyframes/hasFaceTrack/stabilize* are on the same detail rows. Detail rows omit what does not apply: caption styling only on text/subtitle clips, shape styling only on shapes, vector (document metadata, slot overrides — never the document text) only on Lottie/SVG clips, mask only when one is set, keyframes only for animated properties (listed in animated), stabilize/animation blocks only when in use, fade fields only when a fade is set, empty arrays, false booleans and other defaults dropped (absent = default: no fade, speed 1, volume 1, unlinked). Every row carries transform:{x,y,w,h,rotation,opacity} at the playhead, plus layer3d, rotationX, rotationY, z and perspective on a clip switched to a 3D layer (see 3D clip transforms). verbose:true returns the raw untrimmed map
Numbers Every number in a reply is rounded to 3 decimals, except fps and speed-curve pos which keep 6
Validation Toolbox op args are checked against the op's schema before it runs (the homepage tools inspect, capture, frames, activity are not): a missing required key → bad_args naming it; a wrong JSON type → type_mismatch (numeric strings like "1" are still accepted); an enum or declared min/max violation → bad_args listing the allowed values or range. Unknown keys are not errors; they come back as ignored:[…] on success. A clip-ref op with neither clip nor track+index gets bad_args saying so; a stale uuid gets not_found with a hint to re-read inspect
Effects list_effects / list_audio_effects / list_transitions are compact by default (cats:{<cat>:[{id,label}]}); pass id, cat, or q to get parameters. Ids are accepted with . or _ interchangeably; an unknown id returns not_found with the closest matches. Effect stacks live on an adjustment clip linked to the target, created on its own lane the first time; add_effect/add_audio_effect report it as host:{track,index,clip}. Keep addressing the original clip in every effect op, and expect that extra lane in inspect
Images capture, frames, and get_waveform({image:true}) return a text block (JSON meta) followed by an image block. Times are always in the text block; never rely on burned-in labels alone. Sheets fit one vision image (≤1456 px long edge)
Overlap Off by default — place/move snap to gaps unless set_overlap enables overlap; the reply reports requested vs placed. With overlap off, moving several clips toward zero must be sequenced back-to-front. set_ripple / close_gap close holes after a delete
Composites make_composite (selection-based) moves the selected clips into a nested timeline and leaves one composite clip plus a bin item. open_composite({clip}) / ({sequence}) makes that timeline the one every op edits; open_composite({main:true}) returns. Adding content past a composite's end never lengthens placed instances, and shortening the content never shortens them (or the adjustment clips hosted on them) — trim them with set_duration; their trim limit follows the content. set_clip_speed retimes a composite as a unit: on an untrimmed instance, speed = currentDuration / target makes it end at target. set_speed_curve also works on composites. One level only: no composites inside composites. flatten_composite renders the clip to a video file (async, poll export_status) and swaps it in. export_video always renders the main timeline
Export export_video is async — poll inspect().export or export_status. The output path is normalised, so use the path echoed back. Named sizes: list_export_presets + export_with_preset
Atomicity apply is not atomic: on failure the ops before it stay applied. The failure reply is {ok:false, error:"apply_failed", stopped:<index>, tool, failed:<that op's error>, done:[results of the ops that ran]} — done never contains the failed op. An ops array cannot reference an id produced earlier in the same batch — end the batch after set_speed_curve
Undo One batch = one undo step. Linear history (no branches): list_history({limit}) returns the newest limit (default 20) versions as {index, label, short} plus the HEAD hash/short and total (index 0 = Origin). undo_to({index}) or undo_to({hash}) jumps (a ≥8-char prefix such as short works); the next edit drops redo. take_snapshot writes compact project JSON named <hash>.json (file SHA-256 = history hash). Ops outside the stack (and all read-only ops): import_media, import_media_bytes, seek, play, pause, undo, redo, undo_to, take_snapshot, restore_snapshot, set_overlap, set_ripple, set_snap, set_loop_work_area, export_video, save_project, set_theme, set_shortcut, reset_shortcuts, set_beat_layers, detect_beats, list_speed_curve, list_fade_curve, install_addon, cancel_addon_install, set_acceleration, switch_angle, end_multicam, market_download, market_cancel_download, and the preset-store ops rename/delete/export/import_user_text_preset, rename/delete/export_text_animation_preset. set_beat_layers changes the user's own snapping and cannot be undone
Errors {ok:false, error:<code>, detail:<text>} — bad_args, not_found, type_mismatch, unknown_op (the detail suggests the nearest op names and their toolbox), unknown_toolbox, wrong_endpoint, wrong_toolbox, apply_failed, import_failed, import_timeout, export_busy, export_failed, export_timeout, capture_failed, conflict, unsupported (a build without the vector renderer), and from the market toolbox market_unavailable, consent_required, market_error, download_failed. type_mismatch is also the answer when an op gets the wrong kind of clip (a shape to apply_text_preset, a title to set_shape_style, an image to set_clip_orientation)
Change detection Every inspect includes revision; pass since:<revision> to get {unchanged:true} when state is current
Media import Absolute paths or file://, or import_media_bytes (base64 — always pass an explicit path). No directory listing. If the user gave a fuzzy name (GX010023.mp4 in Downloads), glob/search with your own filesystem tools, then pass the hits to import_media and confirm missing:[] is empty

Async jobs

All return {started:true} immediately. Every field below except export lives under inspect({detail:true}).jobs, which lists only the jobs that are running (the whole jobs key is absent when nothing is).

Started by Poll
export_video export.{active, progress} (or export_status)
package_project jobs.package.{active, progress}
generate_subtitles jobs.subtitleGen.{active, progress, status}
set_clip_reverse (video) jobs.reverseRender.{active, progress, status}
market_download jobs.market.active (count of running downloads); per-job status, progress, error, path and asset from market_downloads
detect_scenes jobs.sceneDetect.{active, progress, status, clip, scenes} (also present, inactive, while a scan result is loaded)
run_segmentation, segment_clip, apply_denoise, detect_faces No progress field — re-read inspect({clips:true, detail:true}) and compare
stabilize_clip Per-clip stabilizing / stabilizeProgress / stabilizeStatus on the detail clip row
transcribe, diarize, tts_generate, sfx_generate, estimate_depth Return {job_id} instead: poll get_job({id}) until active is false, then read ok plus result or error. cancel_job({id}) stops one. inspect({detail:true}).jobs.list keeps recent jobs, finished ones included

Toolbox reference

Toolbox When to use
media Import (paths/bytes), list, rename, remove, replace, export still, set_asset_rotation (lossless orientation override for a sideways video)
timeline Tracks, clips, selection, ripple/gap, bookmarks, copy/paste, A/V link
canvas Transform (including 3D tilt, depth and perspective), flip, blend, mask, fade, speed, reverse, animation, stabilisation, set_clip_orientation (lossless 0/90/180/270 for a video clip), shape styling (set_shape_style: the same shading-layer stack captions have — layers / layer patches, add_shape_layer … — plus geometry knobs; the legacy flat fill/stroke keys still land on the fill/stroke layers) — see Shapes
playback Seek, play, pause, In/Out work area
text Title and caption clips, style packs (list_text_presets, user presets), fonts, shading layers (fill/stroke/shadow/glow/extrude), gradient presets and shader effects, looks, In/Out/Loop animation presets, Lottie preset import
shapes Builtin shapes, stickers, emoji
motion Lottie animations and SVG drawings as vector clips: add, inspect, swap the document, re-theme through slots or the svg.* element overrides
model3d 3D models (glTF binary .glb) as model clips: add, inspect, pick the animation, pose and light them — see 3D models
subtitles Subtitle clips, cues, import/export, Whisper generation
effects Video/audio effects, transitions, templates, effect clipboard
project Open/new/save/package, canvas, background, metadata, export
keyframes Property animation keys and tangents — clip transform, fx.<i>.<param>, mask.<key>, on text/subtitle clips text.<key> (pixelSize, letterSpacing, lineHeight, boxPadding, pathBend) or text.layer.<id>.<field> (opacity, offsetX, offsetY, blur, width, spread, trimStart, trimEnd, dashOffset, sketchLength, sketchDeviation, color.r/g/b/a, gradient.angle/offset/scale/center.x/y, gradient.stop.n.pos, effect. for an effect paint's scalar params); the old names outlineWidth, shadowBlur, glowRadius, gradientAngle, color.r… still resolve onto the stroke/shadow/glow/fill layers. On shape clips the same layer fields as shape.layer.<id>.<field> (a fresh shape's layers are fill and stroke) plus `shape.<cornerRadius
speed Speed ramps; reading custom fade curves (write them with set_fade_curve in canvas)
segmentation SAM-style cutout (session or one-shot)
ai Denoise, face detection, auto-reframe, depth estimation for the depth effects (estimate_depth, sample_depth, clear_depth) — see Depth effects, add-on install
audio Waveforms, silence, loudness, ducking, beat detection, beat-synced cuts, clip volume
scene Shot detection, what is in each shot, scene-synced cuts
ui Theme, shortcuts, editor preferences, guides
multicam Multi-camera session: set up, switch at the playhead, save
market Stock media from the Cutwire marketplace: status/consent, search, resolve a link, download into the bin
transcript Word-level transcripts stored on each asset, reading them (get_transcript), cutting by words (cut_words) or source ranges (keep_ranges), building a sequence from an edit list (assemble), speaker labels (diarize) — see Editing speech
voice ElevenLabs / Fish Audio voiceover (tts_generate), ElevenLabs sound effects (sfx_generate), list_voices, cloud_provider_status — see Cloud voices

Working to the music

detect_beats blocks and returns the grid — no polling.

{"name": "detect_beats", "arguments": {"start": 0, "duration": 30}}
{"ok": true, "bpm": 128.0, "confidence": 0.81, "beatsPerBar": 4, "firstDownbeat": 2,
 "beats": [0.31, 0.78, 1.25], "onsets": [{"at": 0.31, "s": 0.9}], "cached": false}

Then either drive edits from those exact times, or arm the grid and let snapping do it:

Call Effect
set_beat_layers({grid:true}) Beats become snap targets — place_clip, move_clip and move_to_track now magnet to the nearest beat within 150 ms, and so do the user's own drags
split_on_beats({clip, unit:"bar"}) Cuts the clip at every bar line. One undo step
snap_clips_to_beats({clips, unit:"beat"}) Quantises clip starts. One undo step
bookmark_beats({unit:"bar"}) Writes the grid into the project as markers that outlive the analysis

The analysis is transient: any edit that changes the mix drops it (finishEdit clears it when the audio fingerprint moves). inspect({detail:true}).beats reports {active, analysed, bpm, confidence, rangeStart, rangeDuration, n, onsets, gridVisible, onsetsVisible, stale} — check stale before trusting a grid you fetched a few ops ago, and re-run detect_beats when it is true.

Seeing the footage

Agents cannot watch video, so the three read tools give a text-then-image path from “where does something happen” to “what is it”:

  1. activity({start, end}) — text only. Samples the composition (default 200 frames across the range, tiny render) and returns three arrays plus the strongest peaks:

    {"ok":true,"space":"timeline","start":0,"end":12,"step":0.06,"n":200,"scan":[64,36],
 "content":[0,1.2,0.9,…],"motion":[0,0.02,…],"audio":[0.41,…],
 "peaks":[{"t":3,"content":69.8},{"t":6,"content":67}],"max":{"content":69.8,"motion":0.6,"audio":0.9}}

    content is the mean HSV change between consecutive samples on a 0..255 scale — the same metric and scale as detect_scenes (27 is a cut at full frame rate). motion is the fraction of pixels that moved, audio the mixed peak, both 0..1. At coarse steps a fast pan scores like a cut, so these are peaks, not cuts.

  2. frames({…}) — one JPEG contact sheet, about the cost of a single capture. Each tile has its index and time burned in, and the same times come back in frames[].t.

    Mode What it renders
    sample:"changes" (default) Up to n (12) visually distinct frames. Candidates are hashed (dHash) at ~4 per second, capped at 120; a frame is kept only if it differs from the last four kept by more than min_change bits (default 12; in-shot motion scores ~5–10, a cut 30+). skipped says how many looked the same; next:{start,end} appears when more distinct frames remained than n allowed
    sample:"uniform" n evenly spaced frames
    sample:"scenes" The detect_scenes thumbnail of every scanned shot in the range (frames[].clip, frames[].scene); unscanned clips contribute their first frame and are listed in unscanned
    at:[…] Exactly those times (max 20)

    Add clip (or track+index) to render one clip's source frames instead of the composition: t is then source seconds and tl the timeline time. return:"path" writes the sheet next to the project's frame captures instead of inlining it. diff on each frame is the dHash distance to the previous kept tile. dur is the length of the material; a tile past it renders black and carries beyond_end:true (the count is repeated at the top level).

  3. capture({at}) — one 1280 px still for detail. beyond_end:true flags a time past the timeline end.

For audio, get_waveform({image:true, …}) returns a PNG with a mixed-peaks lane, a speech-band lane, silent stretches shaded, onset ticks when detect_beats is current, a lane of the transcript's words when the media is transcribed (indices in the reply's words), an optional spectrogram:true lane, and a time axis — together with a summary_buckets (50) numeric summary and the silence ranges it found. In clip mode the image form is timeline-space (through the clip's volume and fades), unlike the numeric clip form.

Typical recipe: activity() → frames({at: peaks}) or frames() → edit → capture({at}).

Editing speech

The cheapest way for an agent to understand talking footage is to read it, not watch it. Transcribe each source once, read the words with their times, cut between words, and look at pictures only at the cuts.

  1. transcribe({clip}) (or {asset}, or {clips:[…]}) transcribes the whole source file behind the clip and stores the result on the asset, in source time. It survives every later trim, split, reorder and undo, and is saved with the project; calling it again returns cached unless force:true. Async — poll get_job.

    engine What you get Needs
    local (default) Whisper words, each placed by a CTC forced aligner (real word start/end), Silero VAD keeping Whisper off silence, speaker labels with diarize:true whisper-model; align-model for the language (word timings, else interpolated and aligned:false); vad-model and diarize-model optional
    elevenlabs Scribe: verbatim (fillers like "um" kept and tagged), audio events like (laughter), speakers with diarize:true, keyterms to bias names The user's key and consent (Settings → Cloud providers). Billable

    Whisper tends to drop fillers, so a local transcript tags only the ones it kept.

  2. get_transcript reads it three ways:

    • {asset}: the whole take in source seconds, which is what keep_ranges and assemble take.
    • {clip}: only what that clip plays, in timeline seconds.
    • {start, end}: everything audible on the timeline in that range.

    The default view:"phrases" breaks lines on a pause of at least break_on_silence (0.7 s) or a speaker change, and adds a compact string, [12.40-15.10] S1 text per line, which is the cheapest read of a long take. view:"words" lists every word with a stable index i into the asset's transcript, plus type (filler/event), speaker, conf and estimated (the aligner could not place it). Page through with offset/limit/next_offset.

  3. Cut:

    • cut_words removes word runs (words:[[i,j]]) or text (text:"um", or match:"phrase" for a phrase). snap:"boundary" also takes the pause around the removed words, keeping padding (0.05 s) of air beside the words that stay. snap:"silence" cuts at the quietest point of each gap instead. dry_run reports the plan without editing.
    • keep_ranges rebuilds one clip from source ranges, in any order: its edit decision list.
    • assemble places [{asset, start, end}] from several sources back to back (the best take of each beat), appended to a track.
    • remove_silence with method:"vad" cuts only where nobody speaks; energy thresholds mistake music and room noise for speech.

    All of these:

    • cut linked video and audio together;
    • ripple what follows;
    • give every cut edge a short audio-only de-click ramp (declick, 0.03 s; it never fades the picture);
    • leave one undo step.

  4. Check each cut. get_waveform({image:true}) draws a lane of the transcript's words under the waveform (the reply's words carry their indices), and frames({at:[…]}) shows the picture on both sides.

  5. generate_subtitles on transcribed media builds captions straight from the stored words, instantly and on the exact word times, however the clips were cut. On media without a transcript it runs Whisper as before.

diarize({clip}) labels speakers on its own (and relabels an existing transcript's words). A butt cut between two clips' audio can take a real crossfade: add_transition works on audio tracks. With media past the cut on both sides it is a true crossfade; a side with no media dips through silence instead.

Cloud voices

The voice toolbox needs the user's own ElevenLabs or Fish Audio account. Keys live in Settings → Cloud providers, or in ELEVENLABS_API_KEY / FISH_API_KEY for headless runs. Nothing is sent until the user also turns on that provider's consent switch.

  • cloud_provider_status says what is ready. Keys are never returned.
  • list_voices({provider}) lists voice ids.
  • tts_generate({provider, text, voice}) and sfx_generate({prompt, duration}) write an MP3 under the app data folder, import it into the bin (the asset records what made it), and with place put it on a free audio lane.
  • Every call is billable. not_configured and consent_required errors say what to ask the user.

Understanding the footage

capture shows one composited frame. The scene toolbox instead builds a structured index of the source material: where every shot begins, how active each one is, and — with the object add-on — what is in it.

detect_scenes is async. Poll inspect({detail:true}).sceneDetect until active is false.

{"name": "detect_scenes", "arguments": {"clip": "…uuid…", "with_objects": true}}

A clip already scanned at the same settings returns {"cached": true} and is ready at once.

Call Effect
describe_clip({clip}) One-call impression: shot count, shortest/longest, mean score, top shots, and labels ranked by screen time. clip (or track+index) reads any scanned clip's cache; omit it for the last scanned clip
list_scenes({clip, sort:"score"}) Every shot, with timeline_start/timeline_end already mapped through trim, speed and reverse, and thumb/timeline_thumb for its representative frame. Takes the same optional clip ref as describe_clip
find_scenes({label:"person"}) Searches every scanned clip on the timeline, best first — this is how you gather material
split_on_scenes({clip}) Cuts at every boundary. One undo step
bookmark_scenes({clip}) Marks the boundaries instead of cutting

Times come back twice. start/end are seconds into the source file; timeline_start/ timeline_end are the same moments on the timeline. Act on the timeline pair — the mapping through trim, speed and reverse is already done for you.

Unlike beats, this analysis is not transient. It describes the source file and is cached against that file's timestamp, so it survives edits, undo and reload. Re-trimming a clip changes the scanned range, so that does scan afresh.

with_objects needs the object-model add-on. Call ai_capabilities to see what is installed and what each piece unlocks; list_addons / install_addon can install a missing model.

Finding files. Drift does not list directories. When the user names a clip loosely (GX010023.mp4 in Downloads, “the wedding file”), glob or search with your own filesystem tools, pass the absolute path to import_media, and treat a non-empty missing:[] as “search again”, not as a bin problem.

Motion graphics

The motion toolbox puts Lottie (Bodymovin JSON) animations and SVG drawings on a graphic track as vector clips: they are drawn by Skia at whatever size the clip box has, follow the clip's speed/reverse, and take effects, masks and transitions like any other clip. Builds without the vector renderer fail every add_* op with unsupported.

Call Effect
import_media({paths}) Also takes Lottie .json files and .lottie bundles (unpacked into app data, one asset per animation); the asset then places like any other with place_clip
add_lottie({json, at, track, duration, fit, loop, offset, slots, name}) json is the document text (inline, ≤ 8 MB) or an absolute .json path. Plays once at its own length unless duration is set; loop (hold default, loop, pingpong, hide) decides what happens past the end; fit (contain default, cover, stretch) how it fills the box. Returns {id, track, index} plus the inspect summary
add_svg({svg, at, track, duration, fit, slots, name}) Same for an SVG still (default 5 s). SMIL animation and scripts are ignored and reported in unsupported. Returns elements:[{id, tag, classes, inDefs, fill, stroke, strokeWidth, opacity}], the ids the svg.<id>.* overrides address
inspect_lottie({json | svg | path | clip}) Read-only. {version, fps, durationSec, width, height, layers, slots, namedProperties, fonts, markers, expressions, unsupported, hints}; for an SVG also elements
set_lottie_source({clip, json | svg | path}) Swap the document; position, length, fit and loop stay, slot overrides survive only where the new document declares the same slot with the same type
set_lottie_options({clip, fit, loop, offset, name}) Playback options; only supplied keys change
set_lottie_slot({clip, name, value}) Override one declared slot; value:null clears (and drops its keyframes). Typed: color "#rrggbb"/"#aarrggbb"/name/[r,g,b,a] in 0..1, scalar number, vec2 [x,y], text string, image path. On an SVG the names are the reserved override keys below
list_lottie_slots({clip}) Declared slots with current overrides; for an SVG the four drawing-wide keys plus the element overrides that are set
get_lottie_source({clip}) The document text (can be large) — edit it and send it back with set_lottie_source when the animation has no slots

Slots are the templating mechanism: an animation exported with slots (Lottie ≥ 5.10, After Effects "Essential Properties") can be re-coloured or re-worded per clip without touching its JSON. capture / frames render vector clips like everything else, so check the result at two times before relying on an animation.

An SVG declares no slots; it is restyled through reserved keys instead. svg.fill, svg.stroke (colour), svg.strokeWidth, svg.opacity (number) act on the whole drawing — drawing-wide colours replace paints the file already has, so a fill="none" outline stays hollow and no stroke is added where the file drew none. svg.<elementId>.<fill|stroke|strokeWidth|opacity|visible> act on one element from elements (ids are case-sensitive; visible is 0/1). Everything but visible keyframes as vector.svg.… — scalars directly, colours per channel (vector.svg.logo.fill.r, one set_keyframe per channel, 0..1). An .svg dropped in the bin (import_media) is a vector asset and places as a vector clip.

3D clip transforms

Every visual clip except a 3D model clip can be made a 3D layer, then tilted and pushed in depth. The switch is per clip (layer3d, off by default; the inspector's "3D layer" switch). Four more transform properties sit beside x/y/w/h/rotation. Writing any of them, through set_transform or set_keyframe, turns the switch on, and a clip loaded with any of them set comes up switched on. set_transform({layer3d:false}) flattens the clip: all four go back to their defaults (keys included; undo brings them back). set_transform writes them at the playhead (the same keyframe rules apply), set_keyframe animates them under the same names, and reset_transform clears them and switches 3D off:

Property Default Meaning
rotationX 0 Degrees; + tips the top edge away from the viewer
rotationY 0 Degrees; + swings the right edge away from the viewer
z 0 Project pixels; − pushes the clip away (smaller), + pulls it toward the viewer
perspective 2000 Distance of the eye from the canvas, in project pixels; smaller = stronger foreshortening

The clip turns about its own centre. Rotations are intrinsic: X, then Y, then the existing rotation, so rotation spins the clip within its tilted plane. The eye sits perspective px in front of the canvas centre, so every clip is seen by the same virtual camera. As a result, a clip off-centre drifts toward the centre as z decreases. At z = −perspective it is half size, and at z ≥ perspective it is behind the eye and draws nothing.

Masks and effects are applied in the clip's own frame before the tilt, so they follow it. Clip intro/outro animations and text animations stay flat, on top of the tilt. A clip whose 3D values are all zero renders exactly as before. inspect and set_transform readouts include layer3d and the four fields only on a 3D layer.

Transform layers

A transform layer moves, scales, rotates, tilts and fades several tracks as one without nesting them in a composite. It is an adjustment track with scope range: it covers every video, text, subtitle and graphic track from just below it down to its span end, and each clip on it is a transform clip (adjustmentKind: "transform") whose ordinary transform is applied as a parent to every clip it covers. Per-clip transforms stay as they are; a covered clip renders as parent × its own transform, so its x/y/w/h are still measured inside the canvas and the layer carries them.

  • make_transform_layer({clips:[ids]}) adds a layer directly above the topmost of those clips' tracks, spanning to the lowest, with one clip over their time range. add_track({type:"transform", span_end?}) adds one at the top instead (default span: every visual track below), with an identity clip over the whole timeline.
  • set_transform on the transform clip moves the group. Its box defaults to the full canvas, which is the identity; shrink w/h to scale the group, set rotation, or layer3d with rotationX/ rotationY/z to tilt it as one flat card. opacity multiplies into every covered clip. set_keyframe and clip intro/outro animations work on it like on any clip.
  • set_transform_span({track, end | end_clip}) changes the span. The end must be a visual track below the layer; it may not cut through a nested layer's span or reach past an enclosing one.
  • Layers nest: a layer inside another's span is itself moved, and composes outer × inner.
  • set_track({hidden:true}) on a layer bypasses it: the covered tracks stay visible, unmoved.
  • inspect rows: adjustment tracks report scope, lanes parent, a layer span_end and covers, and any covered track transformedBy (layer indexes, outermost first). set_transform and reset_transform on a covered clip report parented.

Limits: opacity is multiplied per clip, so overlapping covered clips show through each other (a true group fade still needs a composite); standalone effect adjustments inside the span, and their masks, are not moved; a transition mixes after both of its sides have been moved; and content a clip places off-canvas becomes visible when the group is scaled down.

3D models

The model3d toolbox puts a glTF binary (.glb only — a .gltf with sidecar files would not survive bundling) on a graphic track as a model clip. The clip is a full-canvas layer: the model is drawn by its own camera into it, so it is never clipped by a box edge, and it takes opacity, fades, blend modes, effects, masks and transitions like any other clip. Track order alone decides stacking; depth is perspective, never z-order.

Call Effect
import_media({paths}) Also takes .glb files; the asset places with place_clip like any other (its duration is the first animation's length, 5 s for a static model)
add_model3d({path, at, track, duration, animation, loop, offset, scale, depth, rotX, rotY, rotZ, lightYaw, lightPitch, lightIntensity, ambient, name}) Absolute .glb path. Returns {id, track, index, animations:[{name, durationSec}], vertexCount, warning?, model3d:{…}}
inspect_model3d({path | clip}) Read-only. {animations, vertexCount, primitiveCount, materialCount, textureCount, warning} — warning says what the loader skipped (Draco compression, extra material textures, morph targets)
set_model3d_source({clip, path}) Swap the file; position, length, pose, lighting and keyframes stay, the animation index clamps to the new file
set_model3d_options({clip, animation, loop, offset, scale, depth, rotX, rotY, rotZ, lightYaw, lightPitch, lightIntensity, ambient, name}) Plain (non-keyed) values; only supplied keys change

Placement and pose: the model sits at the clip's x/y (top-left of the full-canvas layer, so 0,0 is centred; move it with set_transform x/y or x/y keyframes — w, h, rotation and the 3D clip transform (rotationX, rotationY, z, perspective) are ignored for this kind, and set_transform still reports the canvas size for them). scale is the fraction of canvas height the model's largest extent spans; depth 0..1 goes from flat (orthographic) to strong foreshortening without changing the on-screen size. rotX/rotY/rotZ are degrees about the model's own axes (intrinsic), applied X, then Y, then Z, each following the earlier ones: X tilts, Y spins about the model's up axis as tilted by X, Z rolls about its forward axis after both. So to spin a tilted globe about its own axis, set rotX for the tilt and keyframe rotY; to stand up a model exported on its side, rotX: -90 then turntable it with rotZ (its original up). Lighting is one key light (lightYaw/lightPitch degrees, lightIntensity) plus ambient 0..1. All nine keyframe as model3d.<key>. Shading is a simple Blinn-Phong on the base colour — normal, roughness and occlusion maps are ignored, so a model reads flatter than in a PBR viewer.

An animated file lists its clips in animations; animation picks one by index, the clip's speed/reverse and offset remap into it, and loop (loop default, hold, pingpong, hide) decides what happens past its end. Node (rigid) animation and skeletal skinning play; morph targets do not (reported in warning). capture / frames render model clips like everything else.

Depth effects

Five effects in the depth category read a clip's estimated depth (relative, 0 = farthest thing in the clip, 1 = nearest, consistent across the clip) and pass the frame through until it exists:

Effect What it does Handy parameters
depth.relight Up to four 3D point/spot lights that fall on the scene, with shadows ambient (how much of the original light remains), light<N>_enabled/x/y/z/color/intensity/radius/cone/aimX/aimY — x/y in 0..1 frame units (may sit off-frame), z −1 (towards camera) … 1 (at the back)
depth.focus Depth of field focusDepth, focusRange, blur (px at 1080p), autoFocus + focusX/focusY to track whatever is at a point
depth.fog Haze thickening with distance fogColor, density, start, ground
depth.view Shows the depth itself (greyscale or colour map) colorize
depth.occlude Goes on a layer above the clip (text, sticker, 3D model): places it at depth inside that clip, so nearer things pass in front target (the clip to sit inside, by id — a string for set_effect_param; "" = the nearest video/image clip beneath), depth, softness, cutoutEdges
  1. ai_capabilities — depth-model must be installed (install_addon otherwise).
  2. estimate_depth({clip, quality}) → {job_id}; poll get_job (it is slow: about 0.5 s per frame on CPU at draft). Re-estimating unchanged pixels returns the cached map at once.
  3. add_effect with one of the ids above on the clip (or, for depth.occlude, on the layer above it — and estimate_depth the clip it sits inside, not that layer) and tune with set_effect_param (set_effect_color_param for light colours) or fx.<i>.<param> keyframes.
  4. sample_depth({clip, x, y, time?}) → {depth} reads what is at a spot — use it to set focusDepth on the subject, or depth on depth.occlude just behind them.

inspect({clips:true,detail:true}) reports hasDepth. The map is cleared when the clip's pixels change (replace source, switch angle, orientation); clear_depth removes it by hand. Standalone adjustment tracks have no depth of their own, so depth effects there pass through.

Text looks and animation

A caption's look is an ordered shading stack (textStyle.layers, layers[0] drawn first): each layer is a fill, stroke, shadow, glow or extrude painted from a solid colour, a gradient (multi-stop; space block|line|word|glyph|accentRun; offset keyframable and offsetSpeed for a moving gradient), a texture (image path) or a shader effect (shine, shimmer, neon-pulse, glitch, chrome, dissolve). Motion is three animation slots — in, out, loop — each a preset id plus params, or an inline After Effects-style animator tree for experts.

Call Effect
list_text_presets({q}) / add_text({text, preset}) / apply_text_preset({clip, preset}) The 33 built-in style packs (title, subtitle, lower-third, caption, quote, impact, pop, neon, handwritten, hormozi, one-word-color, word-background, sentence-background, karaoke-pop, karaoke-highlight, mirage, underline, bulky, word-outline, gold-luxe, chrome, retro-3d, glitch, comic, fire, ice, candy, sticker, rainbow, cinematic, holo-shimmer, sketch, editorial). The list carries each pack's font, accent rule and anim:{in,out,loop} preset ids so you can choose without applying. A pack replaces the whole style — layers and all three animation slots — and is remembered as packId. Unknown ids fail not_found
save_text_preset({clip, label}) / list_user_text_presets / apply_user_text_preset / rename_user_text_preset / delete_user_text_preset / export_user_text_preset({preset, path}) / import_user_text_preset({path}) User packs (user: ids) on disk; apply_text_preset accepts them too. Not project edits, so outside undo
list_gradient_presets / list_text_effects The ready-made paints: 12 gradient stop sets (sunset, ocean, candy, gold, chrome, rainbow, fire, ice, mono, holo, mint, berry — copy stops/kind/angle into a layer's paint.gradient) and the six shader effects with their typed params (shine, shimmer, neon-pulse, glitch, chrome, dissolve — paint:{kind:"effect", effect:{id, params:{<id>:{type, value}}}})
set_text({clip, style}) Partial patch. layers replaces the stack, layer:{id|index,…} patches one, color edits the front-most fill, animation:{in|out|loop:{preset, params, duration, stagger, unit, order, ease, period}}. The flat v6 keys (outline*, shadow*, glow*, fillKind, animIn…) still work and land on the well-known layers
add_text_layer({clip, kind, at}) / set_text_layer / remove_text_layer / move_text_layer / duplicate_text_layer Edit the stack one layer at a time; add_text_layer returns {layerId}. Strokes take strokeAlign center|outside|inside, dash solid|dash|dot|dashdot with dashOffset, trimStart/trimEnd for a write-on and sketchLength/sketchDeviation/sketchSeed for a hand-drawn wobble (fills take the sketch too). The same four tools also edit a shape clip's stack (add_shape_layer … are aliases)
list_text_looks / apply_text_look({clip, look, params}) One-click recipes (Plain, Shadow, Lift, Hollow, Splice, Outline, Echo, Glitch, Neon, Background, Curve, Gradient, Shine, Chrome, Holographic) that rewrite the stack; the style remembers the look, and re-applying the same look with params adjusts just those params (a different look starts from its defaults)
list_text_animations({which, q}) The In / Out / Loop presets with their typed params and flags (unitLocked, orderLocked, easeLocked, durationLocked say which controls the preset ignores; mirrorForOut, mode). Every reveal preset takes duration, stagger, unit (block|character|word|line), order, ease; loops take period (0 = one pass over the clip: hold motion such as tracking-drift) and amount
set_text_animation({clip, which, preset, …}) / clear_text_animation Set or clear a slot. Switching to a different preset resets its params to that preset's defaults unless keepControls:true; durationOverride sets the slot's total length; animators:[…] installs an inline animator tree instead
import_text_animation({path, which}) A Lottie / .lottie text layer's animators (After Effects export) or a Drift preset file, saved as a user preset under the slot's imported category; unsupported lists what was dropped
apply_text_style_to_all({clip, scope}) Copy a caption clip's style to the other subtitle clips on its track (or project)

Reel-style typography, for example: set_text_animation({clip, which:"in", preset:"type-on-blur", stagger:0.05}), set_text_animation({clip, which:"loop", preset:"tracking-drift"}), apply_text_look({clip, look:"gradient", params:{preset:"berry", space:"word"}}).

Every text op fails type_mismatch on a clip that is not a text or subtitle clip.

Shapes

list_shapes → add_shape({shape, at, track}) → set_shape_style({clip, style}). The 29 catalog ids (list_shapes rows carry kind and the default box aspect): basic rectangle, rounded-rectangle, square, circle (an ellipse with a square box), ellipse, triangle, right-triangle, diamond, pentagon, hexagon, octagon, parallelogram, trapezoid; arrows arrow, double-arrow, block-arrow, curved-arrow, chevron; bubbles speech-bubble, speech-bubble-rect, thought-bubble, callout; fun star, lightning-bolt, cloud, heart, cross, burst, banner. An unknown id fails not_found.

style key Read by
kind any — swaps the geometry, keeps the layers
layers / layer the shading stack, exactly as on captions (a fresh shape has fill under stroke; shape strokes default to strokeAlign:"inside"); scope and gradient.space are text-only and do nothing here
cornerRadius native on rounded-rectangle, speech-bubble-rect, callout; rounds the corners of every other kind
points, innerRatio star, burst
headSize arrow, double-arrow, block-arrow, chevron, the banner notch
thickness arrow, double-arrow, chevron, curved-arrow, cross
tailX, tailSize the four bubbles

add_shape_layer / set_shape_layer / remove_shape_layer / move_shape_layer / duplicate_shape_layer are the text layer tools under another name. Animate with set_keyframe({prop:"shape.cornerRadius"}) or shape.layer.<id>.<field>. Looks and style packs are text-only; build a shape's look from list_gradient_presets / list_text_effects.

Stock media from the marketplace

The market toolbox wraps the same service the Assets → Market tab uses (docs/marketplace). It is gated twice: the build must ship a marketplace service (market_status.configured), and the user must have accepted the marketplace terms in the app (market_status.consented). Every other market op fails consent_required until then; there is deliberately no op to accept on the user's behalf, because downloads spend a per-machine quota.

Call Effect
market_status() Types → providers with capabilities (search, featured, resolve), filters (with option ids) and quota, plus account/coins when connected
market_search({q, type, provider, filters, limit}) One page of listings {id, title, type, provider, dur, w, h, coins?, by?, thumb?, variants?} plus offset and has_more; more:true fetches the next page (earlier ids stay valid for market_item/market_download); thumb is a URL you can fetch with your own tools to look at the item
market_resolve({url}) For resolve-only providers (pasted page links); blocks up to 90 s and returns one item
market_item({id}) Variants, preview and license of a listing from the last search/resolve
market_download({id, variant, dir, wait}) Starts the download, spends quota, imports the file into the bin when done and reports its asset id. id must come from the last search/resolve (not_found otherwise); dir must be absolute. wait:<seconds> blocks for completion and turns a failed/cancelled job into {ok:false, error, detail, job}; otherwise poll market_downloads() or inspect({detail:true}).jobs.market
market_downloads({clear}) / market_cancel_download({id}) Job list with status/progress/error/path/asset; cancel a running one

Errors: market_unavailable, consent_required, market_error, and the service's own codes (rate_limited, auth_required, payment_required, provider_unavailable, not_found, download_failed). Coin prices appear only when non-zero; nothing in the reply says "free".

DRIFT_MARKET_API_URL in the environment points the client at another service (tests use a local fake) without rebuilding.

Traps

  • set_transform can write at the playhead. If the property is already animated (two or more keys), or autoKey is on, it creates a keyframe there instead of a constant value. Seek first, or mute the animation with set_property_keyframes_enabled(false).
  • export_video inherits the settings you leave out from the last export made through MCP (agent exports keep their own memory, separate from the export dialog's). Mode switches are the exception — audio_only and gif never carry over — but a codec, a bitrate or a scale does. Pass every setting you care about.
  • A single key is moved, not added to. Visual clips are born with one key at their start on x/y/width/height; with auto-key off set_transform moves that key, so the value holds for the whole clip. Only a property that already has two or more keys (or any property with autoKey on) gets a key at the playhead. Writes are clamped into the clip's own span.
  • auto_reframe cannot make a crop sharper than the canvas. The decode is bounded by the canvas, not by the layout rect, so a crop achieved with an oversized transform resolves at canvas resolution at best. The reply's scale says how far the source is being pushed; upscaled:true means it is past its own resolution.
  • set_mask replaces the whole mask. Omitted keys revert to defaults and omitting shape turns the mask off. Read the current mask from inspect({clips:true, detail:true}).tracks[].items[].mask and send it back merged (its points come as [{x,y}], which the schema accepts alongside [[x,y]]; drop the read-only animated/keyframes).
  • set_subtitle_cues replaces every cue. Read inspect({clips:true, cues:true}) (or a detail row's subtitleCues), merge, send.
  • set_effect_param, set_audio_effect_param and set_transition_param fail not_found on a key the effect does not declare or a stack index that is not there. They used to accept anything and store it silently; a batch that relied on that now stops at the bad op.
  • set_speed_curve returns a new clip id. The old UUID stops resolving. End the apply batch after it — an ops array cannot reference an id produced earlier in the same batch.
  • Keyframe property names are not the transform names: width/height, not w/h (set_transform uses w/h, set_keyframe uses width/height). The 3D names rotationX, rotationY, z and perspective are the same in both.
  • A clip's z does not change stacking. Track order alone decides what draws on top, and tilted clips never intersect. A clip tilted past 90° shows its mirror image; there is no back face.
  • remove_keyframe deletes the nearest key with no distance limit. Confirm the exact time with list_keyframes.
  • set_keyframe_interpolation moves the playhead to at, changing the default time of later ops in the same batch.
  • add_track shifts every track index — index 0 is the new track.
  • import_media_bytes without path writes to a temp dir that is deleted when the call returns, leaving a broken asset. Always pass path.
  • get_waveform reads two different things. With clip or asset it reads the source file, so clip speed, reverse and volume are not applied. Only the timeline form (start + duration, no clip or asset) is what you would actually hear.
  • bpm: 0 from detect_beats is not an error — it means no trustworthy tempo, so beats is empty. The onsets are still valid; pass unit:"onset" anywhere a grid is accepted.
  • Beat analysis dies on the next edit. Read the whole grid out of detect_beats before mutating anything, or re-detect. split_on_beats and snap_clips_to_beats already snapshot it internally.
  • snap_clips_to_beats may not land on the beat. With overlap off, a clip is pushed to the next free gap. Read the to values back rather than assuming they equal the beat time.
  • split_on_beats keeps your clip id for the first piece. The other pieces are new UUIDs, returned in clips in timeline order.
  • There is no track volume. set_volume is per clip; mute a whole lane with set_track({muted:true}).
  • generate_subtitles makes a caption clip with fixed times. Cut first, caption last. From a transcribed asset it is instant and word-exact.
  • Word indices belong to the asset, not the clip. cut_words({words}) takes indices from get_transcript (any view) and they stay valid across cuts; a word the clip no longer plays is skipped.
  • transcribe always covers the whole source file, however short the clip: an hour-long recording is an hour of transcription (and of Scribe billing).
  • keep_ranges refuses speed-ramped clips — a ramp is shaped over the clip's own range. Flatten it (clear_speed_curve) first.
  • One caption clip for a cut-up track. generate_subtitles({clips:[…]}) or generate_subtitles({track, start, end}) transcribes several video/audio clips into a single subtitle clip; the sources must not overlap in time. export_subtitle_file({clip, path, timeline_times:true}) offsets cues by the clip's start so the file matches the exported video.
  • apply_denoise is noise, not reverb. "Sounds like a bathroom" will not be fixed by denoise.
  • activity.content at coarse steps reads pans as cuts. Confirm a peak with frames({at:[…]}) before cutting on it.
  • frames({clip}) times are source seconds. Use tl for the timeline position.
  • get_waveform({image:true}) in clip mode is timeline-space, while the numeric clip form reads the raw source file.
  • Lottie expressions are not evaluated. Skottie renders an expression-driven property at its static value. inspect_lottie (and every add_lottie reply) lists them under expressions; bake them to keyframes in the authoring tool before relying on the motion.
  • set_lottie_slot is typed against the document. An undeclared slot or a value of the wrong type fails bad_args — read the slots array from add_lottie or list_lottie_slots first. Documents with no slots can only be changed by editing the JSON (get_lottie_source → set_lottie_source).
  • add_lottie with a slots map does not fail on a bad slot — it places the clip and reports the rejected ones in slotErrors.
  • set_text.layers replaces, set_text.layer patches. Send the whole array only when you mean to rebuild the stack; layer ids are lowercase and keyframes on a removed layer are dropped.
  • Switching a text animation preset resets its params. Send preset and the tweaks in one set_text_animation call, or add keepControls:true (in set_text_animation, or inside the slot object of set_text.style.animation) to keep duration/stagger/unit/order/ease/period/amount across presets. Re-sending the current preset id keeps everything.
  • apply_text_look with a different look starts from that look's defaults. Only re-applying the clip's current look merges params.
  • Stagger duration is per unit. A word-by-word reveal takes duration + stagger × (words − 1); short subtitle cues can cut it off.
  • list_emoji needs q or group — the catalog is ~1900 entries; add_emoji takes the id (the character).

Example

An ops array is submitted whole, so it cannot reference an id produced earlier in the same batch. Import and place first, then read the new clip's id out of the reply:

{
  "ops": [
    {"tool": "import_media", "args": {"paths": ["/path/to/clip.mp4"]}},
    {"tool": "place_clip", "args": {"asset": "0", "at": 0}}
  ]
}

The reply carries each op's result in order — done[1].result.id is the new clip's UUID. On failure done holds only the ops that ran and failed the error of the one at stopped. Use the id in the next call:

{
  "ops": [
    {"tool": "set_duration", "args": {"clip": "<done[1].result.id>", "duration": 5}},
    {"tool": "set_transform", "args": {"clip": "<same id>", "x": 0, "y": 0, "w": 1920, "h": 1080}}
  ]
}

Then verify visually — capture is a homepage tool and returns unknown_op inside ops, so call it on its own:

{"name": "capture", "arguments": {"at": 2.5}}

Measuring the surface

scripts/mcp-probe.py starts build/drift --headless, synthesises a four-shot clip with ffmpeg, calls every read tool plus a sample of ops, and prints the reply size of each in characters (≈ tokens × 4). --out DIR saves the replies and returned images; --calls FILE runs your own list. Run it after touching the MCP layer to see what an agent will pay.

Security

  • Binds to 127.0.0.1 only.
  • Any local process with the session token has full editor access.
  • Turn Agent access off when finished.

Flatpak: host file import may need flatpak override --filesystem=home org.cutwire.Drift.