AI agents for photo and video editing: the field manual

00:00 · Before you start

The split that works

An agent is excellent at the parts of editing that are exact and repeatable: trimming to a timestamp, setting loudness, laying captions on a grid, exporting six sizes from one master. It is poor at the parts that need eyes and taste. The teams that get good results divide the job on that line and never ask the agent to fake the half it can't do.

The agent never watches the video. It reads it: the transcript, the frames, the numbers.

Any agent that can run commands on your computer works here: Claude Code, Codex, Cursor's agent and the others like them. You need ffmpeg installed (version 8 or later if you want its built-in transcription), and optionally ImageMagick and Node.js for pictures. Ask the agent to check what is installed before it plans anything; a missing filter is the most common reason a plan falls apart halfway.

00:10 · Walkthrough

One ad, start to finish

The project: a 15-second vertical ad for a ceramic travel mug, for TikTok and Instagram Reels, with two extra hook versions to test. Follow the steps in order. Each one says what you do, what to ask the agent, and how to know it worked.

1. Step 1Write the brief

   You: put the brief in the file your agent reads at the start of every session: CLAUDE.md for Claude Code, AGENTS.md for Codex and Cursor (Claude Code reads that one too). Keep it short. Anthropic's own guidance warns that a bloated memory file gets ignored; put rules that apply every session here and leave one-off tasks for the chat.

   FILEAGENTS.mdCopy

   # Project: travel mug ad, fall 2026
   
   Deliverable: 15 s vertical ad, 9:16, 1080x1920, H.264 + AAC,
   30 fps constant, BT.709 tags, faststart. Plus 1:1 and 4:5 stills.
   Platforms: TikTok and Instagram Reels.
   Hook: the product's promise lands in the first 3 seconds.
   Captions: burned in, 2 to 4 words per line, bold sans,
     inside the safe zone (14% top, 35% bottom, 6% sides clear).
   Sound: voice first; music ducked under it. -14 LUFS integrated,
     true peak at or below -1 dBTP, 48 kHz.
   Brand: ink #17160f, signal #c8361a, logo in brand/logo.svg.
   
   Rules
   - raw/ is read-only. Never write, move or delete anything in it.
   - Every render is a new file in work/ or exports/. Use ffmpeg -n.
   - Write each step as a script in scripts/ so it can run again.
   - Before a render, show me the plan or the cut list and wait.
   - After a render, prove it: print ffprobe values and make a
     contact sheet. "Looks done" is not done.

   Check: ask the agent to restate the deliverable in one line. If it gets the size, length or loudness wrong, the brief is unclear, and now is the cheap moment to fix it.

2. Step 2Set up the project folder

   Ask the agent: "Create this folder layout, start a Git repo that tracks scripts and notes but not media, and tell me what ffmpeg filters are available."

   travel-mug-ad/ ├── AGENTS.md # the brief ├── raw/ # originals; read-only │ ├── yerk/ # downloads from Yerk.app │ └── audio/ ├── work/ # every intermediate render ├── exports/ # finished files only ├── scripts/ # one script per step ├── brand/ # logo.svg and fonts ├── prompts.md # every prompt, mode and result └── cuts.json # the approved cut list
   R0Start the projectCopy

   # The folders from the tree above
   mkdir -p raw/yerk raw/audio work exports scripts brand
   
   # Track the thinking, not the gigabytes
   git init
   printf 'raw/\nwork/\nexports/\n*.mp4\n*.mov\n*.wav\n' > .gitignore
   
   # Which filters does this ffmpeg have?
   ffmpeg -hide_banner -filters | grep -E ' (subtitles|loudnorm|whisper|zscale) '

   Why Git for scripts only: the scripts, the brief, prompts.md and cuts.json are small and are the real record of the edit. Git warns at 50 MB per file and GitHub refuses files over 100 MB, so media stays out and gets its own backup.

   Check: the filter list includes subtitles and loudnorm. If subtitles is missing, your ffmpeg was built without libass; on a Mac the standard Homebrew build lacks it and the ffmpeg-full formula has it.

3. Step 3Plan the ad with the agent

   Ask the agent: "Write a 15-second script with the promise in the first 3 seconds, a four-shot list as a table with timings, and one picture prompt per shot. Then write two more hooks that differ in idea, not just wording. Save it all to prompts.md."

   Platform research backs this shape. TikTok's creative guidance asks for the proposition in the first 3 seconds and recommends running 3 to 5 clearly different creatives per ad group. Google's guidance for Shorts action ads favors 10 to 30 second vertical video with sound. Ask for variants that change the idea (the problem, the proof, the payoff), because three rewordings of one hook test nothing.

   OUTprompts.md, as the agent might write itCopy

   | # | Time      | Shot                        | Caption        |
   |---|-----------|-----------------------------|----------------|
   | 1 | 0.0-3.0   | Mug on a car dashboard at   | STILL HOT.     |
   |   |           | dawn, steam rising          | 6 HOURS LATER. |
   | 2 | 3.0-7.0   | Close-up, lid clicks shut   | ONE-HAND LID   |
   | 3 | 7.0-12.0  | Hiker pours coffee on a     | NO SPILLS.     |
   |   |           | ridge, morning light        | NO LEAKS.      |
   | 4 | 12.0-15.0 | Mug on white, logo space    | (end card)     |
   
   Shot 1 prompt: vertical 9:16 photo, matte charcoal ceramic travel
   mug on a car dashboard at dawn, thin steam curling from the lid,
   windshield bokeh of a highway, warm low sun, shallow depth of
   field, empty lower third, no text.

   Check: read the hooks aloud. If the first three seconds don't say what the product does for someone, send it back now; it costs nothing yet.

4. Step 4Make the footage in Yerk.app

   You: open Yerk.app and make each shot from the agent's prompts. This is the part an agent hands to you, and it is quick.

   • Start with pictures. Make each shot as a still in at 9:16, pick the best one, then send it to to become a clip. A picture you approved gives a far more predictable clip than a text prompt alone.
   • Keep the product consistent. Attach your real product photos as references, up to nine, so the mug looks the same in every shot.
   • Make it vertical at the source. Choose 9:16 when you create. Cropping a landscape 1080p clip to vertical leaves about 600 pixels of real width, stretched to 1080.
   • Leave the type out. Ask for an empty lower third and no text. The agent adds exact captions and the logo later.
   • Need sound or length? Several models add sound, and and Seedance 2.5 make clips up to 30 seconds. To change footage you already shot, such as day to night or a new setting, use Edit Any Video.
   • Stuck on wording? Paste the shot into Yerk AI, the built-in chat helper, and it suggests a mode and tightens the prompt.
   • Record the voice and pick the music. Read the script aloud in a quiet room (a phone voice memo is fine) and save it as raw/audio/voice.wav; the agent can convert a memo with ffmpeg -i memo.m4a raw/audio/voice.wav. Add a music track you are licensed to use in ads as raw/audio/music.wav.

   Download each keeper straight into raw/yerk/ with a numbered name (shot-01-a.mp4), and add the prompt and mode you used to prompts.md. Yerk deletes creations after 7 days, so the download is the archive. Once everything is in, lock the folder.

   LOCKMake the originals read-onlyCopy

   # Read-only for every program, not just the agent
   chmod -R a-w raw
   
   # Adding a file later: unlock one folder, add it, lock again
   chmod u+w raw/yerk && mv ~/Downloads/shot-02-b.mp4 raw/yerk/ && chmod a-w raw/yerk

   Why the chmod matters: a "never touch raw/" line in the brief is advice, not a lock. Claude Code's docs say its file rules don't reach scripts the agent runs, and its rewind doesn't track files changed by shell commands, which is exactly how ffmpeg writes. A read-only folder is the one rule every process obeys.

   Check: every shot in the table has a file, every file has a line in prompts.md, the voice and music are in raw/audio/, and touch raw/test fails.

5. Step 5Let the agent look before it cuts

   Ask the agent: "Run ffprobe on every file in raw/ and give me a table of size, frame rate, duration and color transfer. Then make a contact sheet for each clip and look at them."

   This is the agent's version of watching dailies. The ffprobe table catches the two problems that wreck an edit later: variable frame rate (phones do it; the two frame-rate fields disagree) and HDR footage (an iPhone clip whose transfer reads arib-std-b67 or smpte2084 turns grey if it isn't tone-mapped). Recipe R1 makes the table and the sheets; R2 rescues HDR clips.

   Check: the agent describes each clip in a sentence from its contact sheet. If its description doesn't match what you made, it's looking at the wrong file.

6. Step 6Approve a cut list, then render once

   Ask the agent: "Write cuts.json with each segment's source file, in and out times and the reason. Show it to me. Don't render until I say go."

   A cut list is cheap to read and cheap to change; a render is neither. Once you approve it, the agent renders every segment in one pass from the originals (recipe R3), with tiny 30 millisecond audio fades at each cut so nothing pops. One pass matters: every extra lossy re-encode softens the picture a little more.

   For talking footage, ask the agent to place cuts in measured silence rather than on transcript word edges. Whisper's word timings can drift by around half a second, enough to clip the first or last word.

   Check: the rough cut's duration equals the sum of the segments in cuts.json. The agent should print both numbers.

7. Step 7Mix the sound and set loudness

   Ask the agent: "Duck raw/audio/music.wav under raw/audio/voice.wav, normalize the mix to -14 LUFS integrated and -1 dBTP with a two-pass loudnorm, then lay it under the rough cut as work/edit.mp4. Print the measured and final values."

   Two numbers matter and ffmpeg's defaults get both wrong for social. A bare loudnorm aims for -24 LUFS, a broadcast level that sounds quiet in a feed, and it only runs its accurate linear mode when you feed back all four measured values from a first pass. Recipes R4 and R5 do it properly; the clips' own sound is replaced by the mix.

   Check: the second pass reports normalization_type: linear and an output close to -14. If it says dynamic, a measured value didn't get passed through.

8. Step 8Add captions inside the safe zone

   Ask the agent: "Transcribe work/edit.mp4 to SRT, keep the words exactly as spoken, style it from the brief, then render one preview frame with the caption and check it sits above the bottom 35 percent."

   The bottom of a vertical video belongs to the app: the caption, the buttons, the sound name. Meta's Reels ad spec keeps 14 percent at the top, 35 percent at the bottom and 6 percent on each side clear. TikTok publishes downloadable safe-zone template files instead of fixed numbers; have the agent lay the template over a rendered frame and look. Recipe R6 transcribes and previews one frame; the captions are burned in during the final export, so the picture is encoded only one more time.

   Check: compare the SRT with the script. Agents like to tidy grammar, and a caption that says something the voice didn't is worse than none.

9. Step 9Export the deliverables

   Ask the agent: "Export the final 1080x1920 file to the brief's spec, then the two hook variants, then 1:1 and 4:5 stills from the best frame, with the logo placed from brand/logo.svg."

   Recipe R7 burns in the captions and makes the delivery encode in one pass, following YouTube's published upload advice (H.264 High, 4:2:0, closed GOP, BT.709, the index at the front of the file), which every short-form platform accepts. For the hook variants, the agent repeats Steps 6 to 9 with each variant's cut list. The stills come from one master in a single loop (recipe P2), so a later logo change means one rerun, not twelve exports.

   Check: file names say what they are: mug-hookA-1080x1920.mp4, not final2.mp4.

10. Step 10Run the final checks

    Ask the agent: "For every file in exports/, print size, duration, frame rate, pixel format, color tags and loudness, and compare each with the brief. Make one contact sheet of all the variants. Then transcribe the final audio and diff it against the script."

    That last step comes from people who learned it the hard way: re-transcribing the finished mix catches a clipped first word that no frame or number will show. Then you watch the cut yourself, with sound, on a phone. That part stays yours.

    Check: every row matches the brief. Anything that doesn't is a line in the agent's next instruction, not a vague "it looks off".

01:00 · Reference

Video recipes

The recipes follow the walkthrough in order, and each one reads the file the one before it wrote. Every ffmpeg command here except the transcription line was run on test footage before it was printed. Hand them to your agent as starting points; it will adapt the file names. The -n flag makes ffmpeg refuse to overwrite an existing file, which is the safe default for an agent.

# What is this file, really? Two frame rates that disagree = variable frame rate.
ffprobe -v error \
  -show_entries stream=codec_name,width,height,r_frame_rate,avg_frame_rate,pix_fmt,color_transfer:format=duration \
  -of json raw/yerk/shot-01-a.mp4

# One frame a second, 5 across, 4 down: one small sheet per 20 seconds of clip
ffmpeg -n -i raw/yerk/shot-01-a.mp4 \
  -vf "fps=1,scale=240:-2,tile=5x4" work/sheet-shot-01-%02d.jpg

# For HLG clips (color_transfer = arib-std-b67): tone-map to standard BT.709
ffmpeg -n -i raw/phone/clip.mov -vf "\
zscale=tin=arib-std-b67:pin=2020:min=2020_ncl:rin=tv:t=linear:npl=100:p=2020:m=2020_ncl,\
format=gbrpf32le,zscale=p=709,tonemap=tonemap=hable:desat=0,\
zscale=t=709:m=709:r=tv,format=yuv420p" \
  -c:v libx264 -crf 18 -c:a copy work/clip-sdr.mp4

# Two segments from one source, 30 ms audio fades at every cut
ffmpeg -n -i raw/yerk/shot-03-a.mp4 -filter_complex "\
[0:v]trim=start=1.0:end=4.2,setpts=PTS-STARTPTS[v0];\
[0:a]atrim=start=1.0:end=4.2,asetpts=PTS-STARTPTS,afade=t=in:d=0.03,afade=t=out:st=3.17:d=0.03[a0];\
[0:v]trim=start=9.5:end=14.0,setpts=PTS-STARTPTS[v1];\
[0:a]atrim=start=9.5:end=14.0,asetpts=PTS-STARTPTS,afade=t=in:d=0.03,afade=t=out:st=4.47:d=0.03[a1];\
[v0][a0][v1][a1]concat=n=2:v=1:a=1[v][a]" \
  -map "[v]" -map "[a]" -c:v libx264 -crf 18 -c:a aac work/rough-cut.mp4

# A single quick trim: -ss before -i is fast, and re-encoding keeps it exact
ffmpeg -n -ss 00:00:03.20 -i raw/yerk/shot-02-a.mp4 -t 4 \
  -c:v libx264 -crf 18 -c:a aac -b:a 192k work/shot-02-trim.mp4

# The voice drives a compressor on the music, then both are mixed
ffmpeg -n -i raw/audio/voice.wav -i raw/audio/music.wav -filter_complex "\
[0:a]asplit=2[vo][key];\
[1:a][key]sidechaincompress=threshold=0.03:ratio=8:attack=20:release=350[bed];\
[vo][bed]amix=inputs=2:duration=first:normalize=0[mix]" \
  -map "[mix]" -ar 48000 work/mix.wav

# Pass 1: measure. The JSON at the end holds the four values pass 2 needs.
ffmpeg -hide_banner -i work/mix.wav \
  -af loudnorm=I=-14:TP=-1:LRA=11:print_format=json -f null -

# Pass 2: apply, feeding back measured_I, _TP, _LRA and _thresh from pass 1
ffmpeg -n -i work/mix.wav \
  -af loudnorm=I=-14:TP=-1:LRA=11:measured_I=-23.8:measured_TP=-16.4:measured_LRA=2.5:measured_thresh=-33.8:linear=true:print_format=json \
  -ar 48000 work/mix-loud.wav

# Replace the clips' sound with the finished mix; the picture is copied, not re-encoded
ffmpeg -n -i work/rough-cut.mp4 -i work/mix-loud.wav \
  -map 0:v -map 1:a -c:v copy -c:a aac -b:a 192k -shortest work/edit.mp4

# ffmpeg 8+ built with whisper.cpp can transcribe by itself (model file from whisper.cpp)
ffmpeg -i work/edit.mp4 -vn \
  -af "whisper=model=models/ggml-base.en.bin:language=en:destination=work/captions.srt:format=srt" \
  -f null -

# One frame at 2 seconds with the styled caption. -ss goes after -i here, so the
# caption timing matches the frame; before -i the clock restarts at zero.
ffmpeg -n -i work/edit.mp4 -ss 2 \
  -vf "subtitles=work/captions.srt:force_style='FontName=Helvetica,FontSize=11,Bold=1,Outline=2,Alignment=2,MarginV=110'" \
  -frames:v 1 work/caption-check.png

# Captions, color tags and the final encode in one pass
ffmpeg -n -i work/edit.mp4 \
  -vf "subtitles=work/captions.srt:force_style='FontName=Helvetica,FontSize=11,Bold=1,Outline=2,Alignment=2,MarginV=110',\
setparams=color_primaries=bt709:color_trc=bt709:colorspace=bt709:range=tv,format=yuv420p" \
  -c:v libx264 -profile:v high -preset slow -crf 18 -maxrate 12M -bufsize 24M \
  -fps_mode cfr -r 30 -g 15 -bf 2 \
  -c:a copy -movflags +faststart exports/mug-hookA-1080x1920.mp4

# Prove it
ffprobe -v error -show_entries stream=width,height,pix_fmt,color_primaries,color_transfer,r_frame_rate,sample_rate -of compact exports/mug-hookA-1080x1920.mp4

01:20 · Reference

Picture recipes

Most picture mistakes agents make come from three silent defaults: phone photos stored sideways with a rotation note, colors stored in a wider space than the web uses, and metadata that one tool keeps and another throws away. Order fixes all three: rotate, convert the color, then strip.

# ImageMagick 7: the command is magick, never "convert" or "magick convert"
magick raw/photos/IMG_0412.HEIC -auto-orient \
  -profile "/System/Library/ColorSync/Profiles/sRGB Profile.icc" \
  -strip -quality 90 work/IMG_0412.jpg

// sizes.mjs (Node.js with sharp): run with  node sizes.mjs
import sharp from 'sharp';

const master = 'work/master.jpg';
const sizes = [
  ['feed-4x5', 1440, 1800],     // Meta feed, recommended size
  ['square', 1080, 1080],
  ['story-9x16', 1080, 1920],
  ['display', 1200, 628],      // Google responsive display, landscape
];

for (const [name, w, h] of sizes) {
  await sharp(master)
    .autoOrient()                                  // sharp does not do this by default
    .resize(w, h, { fit: 'cover', position: 'attention' })
    .jpeg({ quality: 88, chromaSubsampling: '4:4:4', mozjpeg: true })
    .toFile(`exports/${name}-${w}x${h}.jpg`);
}

// overlay.mjs: render real type to a transparent PNG the size of the picture, then lay it on top
import { chromium } from 'playwright';
import sharp from 'sharp';

const browser = await chromium.launch();
const page = await browser.newPage({ viewport: { width: 1080, height: 1920 } });
await page.setContent(`<body style="margin:0;background:transparent">
  <h1 style="position:absolute;top:300px;left:65px;font:800 96px Archivo;color:#fff">
    Still hot.<br>6 hours later.</h1></body>`);
await page.screenshot({ path: 'work/overlay.png', omitBackground: true });
await browser.close();

await sharp('work/shot-01.jpg')
  .composite([{ input: 'work/overlay.png' }, { input: 'brand/logo.svg', gravity: 'south' }])
  .jpeg({ quality: 90, chromaSubsampling: '4:4:4' })     // keep type edges sharp
  .toFile('exports/shot-01-titled.jpg');

# Every export on one labelled sheet the agent (and you) can review at a glance
magick montage -label '%f' exports/*.jpg -tile 4x -geometry 360x360+8+8 work/review-sheet.jpg

# Cut-out product shots: choose a model whose license fits commercial work
rembg i -m birefnet-general raw/photos/mug.jpg work/mug-cutout.png

01:40 · Reference

Gotchas agents fall for

These are the problems that come up again and again in practitioners' write-ups. Paste the list into your brief, or ask the agent to read this page first.

1. Phone footage with a wandering frame rate

   iPhones record variable frame rate, which editing software can misread and which drifts out of sync. Convert to constant with -fps_mode cfr -r 30 before editing.

2. Joining clips that don't match

   ffmpeg's concat demuxer needs every file to share codecs and time base. Mixed sources go through the concat filter, which re-encodes, as in recipe R3.

3. Old tutorial code

   MoviePy 2 removed moviepy.editor and renamed every set_ method to with_. Agents trained on older examples write the old calls; tell them which version you have.

4. Timeline export that won't install

   The OpenTimelineIO core package no longer ships the EDL and Final Cut XML adapters; they live in OpenTimelineIO-Plugins. Install both when an editor will finish the cut in Premiere, Resolve or Final Cut.

5. Silence cutting that keeps the flubbed take

   auto-editor cuts by loudness, so a fluent wrong take survives. Tell the agent your habit ("the last take of each line is the keeper") and have it cut from the transcript.

6. A silence threshold that does nothing

   One creator's hard-coded -35 dB threshold missed every pause on a recording whose speech sat around -24 dB. Measure each recording's levels first and set the threshold from them.

7. Passing unit tests, wrong edit

   A cut engine with 85 green unit tests still cut real footage wrongly. Media needs media checks: frames, durations and a re-transcription of the output.

8. Pillow dropping the color profile

   Pillow saves a JPEG without its ICC profile unless you pass icc_profile=im.info.get('icc_profile'). The colors shift and nothing errors.

9. Stripping the one tag a store requires

   Google Merchant Center asks that AI-made product images keep their IPTC trainedAlgorithmicMedia tag, and its listing images allow no added text or logos. A "strip everything" step can put a listing out of policy.

10. Licensed tools at company scale

    Remotion, the React video framework with official agent skills, is free for individuals and companies of up to three people; larger companies need a company license.

11. Licensed music on long Shorts

    YouTube blocks a Short longer than one minute worldwide if it carries an active Content ID claim. Keep Shorts with licensed music at 60 seconds or less.

02:00 · Reference

How the agent checks its own work

Anthropic's guidance for Claude Code puts it plainly: the agent stops when the work looks done, and without a check it can run, looking done is the only signal it has. For media, give it four.

02:20 · Reference

Delivery spec sheet

Published platform requirements as of October 2026. Give the row you need to the agent as its acceptance criteria.

Loudness: no major platform publishes an official target; about -14 LUFS integrated and -1 dBTP true peak is common practice. Audio at 48 kHz AAC follows YouTube's upload advice.

02:40 · Reference

Instructions an agent can finish

A good media instruction reads like a delivery spec: what goes in, what comes out, how to prove it. Compare these two.

Make this into a TikTok and make it pop.

Cut work/rough-cut.mp4 to a 15 s vertical ad using cuts.json.
Output: exports/mug-hookA-1080x1920.mp4, H.264 + AAC, 30 fps
constant, BT.709, faststart, -14 LUFS, -1 dBTP.
Captions from the script, 2 to 4 words per line, above the bottom
35 percent. Do it on hook A first and show me before the others.
When done: print ffprobe values, make a contact sheet with a frame
at every cut, re-transcribe the audio and diff it against the
script. Don't overwrite anything; use ffmpeg -n.

• One sample before the batch. Anthropic's advice for repeated jobs: try two or three files, fix the instruction, then run the rest.
• Plan first when it's big. Claude Code's plan mode (Shift+Tab) lets you edit the plan before anything runs. Codex can start in a read-only sandbox for the same reason.
• Make it a skill once it works. A recipe you repeat weekly, such as "cut a TikTok version", belongs in a skill file the agent loads when needed, which keeps the brief short.
• Two strikes, fresh start. If you've corrected the agent twice on the same thing, clear the conversation and write a better instruction instead of a third correction.

03:00 · Reference

Field notes from practitioners

What people who edit with agents every day have published in 2026.

• Anthropic's growth marketing team built a Figma plugin and a command that produce ad variants and check copy against brand-voice rules, taking ad copy from about 30 minutes to about 30 seconds. A person still reviews the output before upload. (Claude blog, January 2026.)
• The browser-use team's video-use project sums up the approach in its README: the model never watches the video, it reads it. It looks at filmstrip-and-waveform images only at decision points, fades audio for 30 milliseconds at every cut, and checks every cut in the render before showing anything. (April 2026.)
• LaoZhang's test edit found Whisper ending a word half a second early; snapping cuts to measured silence kept every word. The same write-up found Homebrew's standard ffmpeg had no subtitle filter. (15 October 2026.)
• Gen Furukawa built a transcript-editing skill that shows a review page of every removed phrase before rendering, then re-transcribes the preview. That re-transcription caught a clipped first word. (SuperMarketers, 8 October 2026.)
• Mahnoor Faisal cut a 9-minute take to 1:22 with an agent. The captions needed no fixes, but cutting took much of an afternoon, and rapid repeated words had to be re-transcribed. (XDA, 15 July 2026.)
• Hasan Aboul Hasan's editor writes cuts to a JSON file, runs a verification script on every render and uploads only as a private draft for a person to approve. (July 2026.)

The common thread: agents save the most time on the exact, repeated work, and every successful setup keeps a person approving the cut and watching the result.