Automatic Checks Overrate the Cheapest AI Video Model: A Pre-Registered Cost and Quality Study

1Methods

1.1 Material and shots

Footage came from Faithful in Little, a three-part 9:16 faith drama. Four shots were chosen to cover the main shot types of narrative coverage. Every condition started from the same approved first-frame still (Nano Banana 2.1, 3 credits, counted in every price).

1.2 Conditions

Economy: P-Video. Balanced: Grok Imagine Video 1.5. Premium: Seedance 2.5, planned on D and T. All clips: first frame = the approved still, no end frame, 4 seconds, 720p, 9:16, sound on, one prompt per shot identical across conditions. Two replicates per cell (seeds 101401 and 101402), 16 Economy and Balanced clips in total. The Premium arm could not run as registered and was replaced by observational data (see Deviations).

1.3 Automatic outcomes (pre-registered)

1. Word error rate on D and T: whisper.cpp small.en transcript against the scripted line.
2. Burned-in text: Tesseract OCR on 8 frames, every positive checked by eye.
3. Model-inserted cut: ffmpeg scene score above 0.4 after frame 0.
4. First-frame fidelity: SSIM between frame 0 and the input still.
5. Auto-usable: no burned-in text, no inserted cut and, on dialogue, a word error rate of 0.
6. Cost per finished second: (credits charged + 3 for the still) ÷ 4 s, at the Creator plan rate of $0.025 a credit. Cost per usable second divides that by the usable rate.

1.4 Human rating

One rater (the operator) scored all 16 Economy and Balanced clips from 1 to 5 on realism, motion, identity, artifacts (5 = none) and, for dialogue, lip-sync, plus usable yes or no. Clips were shown by letter only, at full resolution; the key was applied after all 16 ratings were saved.

1.5 Analysis

Means with 95% percentile bootstrap intervals (10,000 resamples); exact binomial intervals for rates. With two clips per cell, intervals are reported only at the condition level, and per-shot values are descriptive. No p-values.

2Automatic results

By the automatic checks alone, the cheapest tier looked good enough for most shots: P-Video passed every check on the dialogue single and the silent wide. It failed the camera move, replacing it with a hard cut in 2 of 2 clips. Grok Imagine Video 1.5 passed all 8 clips, but invented spoken lines on the silent move shot (2 of 2), so its audio must be muted there. Burned-in text appeared in 0 clips; four OCR flags were all false on inspection.

Table 1. Means with 95% bootstrap intervals. Rates are fractions of clips. Premium clips are long multi-line takes from the original production, so the Premium column is not a matched comparison.

Per shot (auto-usable, of 2)

3Human rating reverses the automatic result

The rater accepted 8 of 8 Grok Imagine Video 1.5 clips (exact 95% interval 63% to 100%) and 3 of 8 P-Video clips (9% to 76%). The automatic checks had passed 5 Economy clips. Priced per human-usable second, Balanced costs $0.27 and Economy $0.38: the cheaper model per second is the dearer one per usable second.

Table 2. Blinded operator rating, 16 clips. Bootstrap 95% intervals for means, exact binomial for usable rates.

By shot (mean rating; usable of 2)

What the rater saw

P-Video audio was described as grainy or “crunchy, digitized” on 3 clips, and one dialogue take had an abrupt head turn. On the church camera move both P-Video clips broke the pews and seating. Grok clips were described as clear-voiced with realistic motion; its one recurring fault was adding an unscripted extra person on the move shot (2 of 2, both still rated usable).

The automatic checks catch burned-in text, inserted cuts and wrong words. They do not see audio texture, scene coherence or where bodies are placed. Automatic gates alone should not route production.

4Hypotheses

5What to use

The revised routing makes Balanced the default. Prices are FXI Studio user prices on the Creator plan; Visionary is about 0.56× these.

Open the Balanced templateOpen the Economy template

6Limitations

• One rater, who is also the publisher. A second, independent rater is needed before citing the per-shot figures.
• Two clips per cell. One film, one genre, four shot types. With n = 8 a single clip moves a rate by 12.5 points, and every interval is wide.
• Imperfect blinding. The rater may have recognised a model’s audio signature; one note named it.
• Premium is observational and unmatched. Its clips used different prompts, inputs, durations (4 to 26 s) and cut structure, so they cannot show what Seedance 2.5 would do on these four shots.
• Automatic measures are proxies. Word error rate mixes recognition errors with real line errors; SSIM on frame 0 measures fidelity to the still, not realism; OCR raised false alarms; neither cut rule is perfect.
• Prices change. All prices are FXI Studio user prices on 14 October 2026 (Creator $0.025 a credit, Visionary $0.014).
• No face processing. No face-recognition or face-embedding model was used and nothing was edited in post.

7Deviations from the pre-registration

1. Premium arm replaced by observational data. The four planned Seedance 2.5 clips failed at dispatch because the model does not accept a seed; one identical retry also failed, and a third call without the seed hit a spend-window limit. None were charged. The arm instead uses the original production’s 9 Seedance 2.5 generations at the credits they were charged.
2. Homophone-normalised word error rate (post hoc). Whisper writes the scripted “brakes” as “breaks”, which failed every D clip under the strict rule. Both measures are reported.
3. First-frame SSIM with best registration (post hoc). P-Video stretches the still instead of cropping it; registered the way the model used the image, its SSIM reads 0.95 instead of 0.78.
4. Frame-difference cut detector (post hoc). The scene-score rule missed three cuts that are plain to the eye. A cut is now any frame whose mean grey-level change exceeds 25/255.
5. Unscripted speech on silent shots (added after seeing the data). It surfaced when Grok invented lines on shot M.
6. Not measured: generation latency, and whether the models honour seeds.

Total spend: 540 credits against a 1,200-credit cap (16 study clips at 480 credits, plus one 60-credit re-roll outside the study).

8Data and code availability

Released under CC BY 4.0. Video files are not redistributed.

• generations.csv: every clip, every automatic measure, user price.
• summary.json: condition and cell summaries with intervals.
• human_ratings_summary.json: all 16 blinded ratings and their summaries.
• Pre-registration: the protocol as committed at 2026-10-14 13:22:15 UTC.
• Contact sheet, shot T and shot M.

The analysis code (metrics, cut detection, bootstrap, ratings) is kept in FXI Studio’s repository and is available on request from the authors.

9How to cite

APA

FXI Studio Research. (2026, October 14). Automatic Checks Overrate the Cheapest AI Video Model: A Pre-Registered Cost and Quality Study (Version 1.1). FXI Studio. https://www.fxi.studio/research/ai-video-cost-quality-study

BibTeX

@techreport{fxi2026costquality,
  title       = {Automatic Checks Overrate the Cheapest AI Video Model: A Pre-Registered Cost and Quality Study},
  author      = {{FXI Studio Research}},
  institution = {FXI Studio},
  number      = {fil-model-study-2026-10-14},
  year        = {2026},
  month       = oct,
  note        = {Version 1.1. Pre-registered 2026-10-14 13:22:15 UTC, commit 3186a4c},
  url         = {https://www.fxi.studio/research/ai-video-cost-quality-study}
}

Version history

• 1.1 · 14 Oct 2026 · Blinded human ratings added; they reverse the automatic headline.
• 1.0 · 14 Oct 2026 · Automatic outcomes, Premium observational arm.
• 0.0 · 14 Oct 2026, 13:22 UTC · Protocol pre-registered.

Related: AI video model benchmark 2026 (speed and success rate on production jobs) and the AI video cost calculator.