# Pre-registration: Cost vs. visual quality of AI video model tiers on a short drama

**Study ID:** `fil-model-study-2026-10-14` · **Author:** FXI Studio · **Registered:** 2026-10-14, before any study clip was generated (commit hash and time = this file's first commit on `research/fil-model-study`).
**Conflict of interest:** FXI Studio publishes this study and sells every model tested here. Read the results with that in mind.

## Questions
1. For each type of shot in a short live-action-style drama, which model tier gives a usable clip at the lowest cost per finished second?
2. How much do the Balanced and Premium tiers add over the Economy tier, and on which shot types?

## Hypotheses (directional, tested as written)
- **H1 (primary, needs human ratings):** a routing that uses the Economy tier on every shot type except where the automatic checks fail reaches ≥ 90% of the Premium tier's mean operator rating at ≤ 25% of its cost per finished second.
- **H2:** Economy's auto-usable rate (definition below) is ≥ 75% across all 8 Economy clips.
- **H3:** on dialogue shots, Balanced's word error rate is ≤ Economy's.
- **H4:** Premium costs ≥ 6× Economy per finished second and is not more often auto-usable than Economy on the two shots where both run.

## Design
- **Footage:** the "Faithful in Little" series (three-part faith drama, 9:16), economy rebuild. The approved first-frame stills from that rebuild are used unchanged in every condition (Nano Banana 2.1 stills, 3 cr each, already made).
- **Shots (4 types):**
  | Code | Shot | Type | Scripted line |
  |---|---|---|---|
  | D | P2 S06, Jax single | single-speaker dialogue | "If those brakes go, it's on both of us." |
  | T | P1 S07, hand on shoulder | two-person touch + line | "He that is faithful in that which is least..." |
  | M | P3 S11a→b, pew pull-back and rise | big camera move, silent | — |
  | E | P2 S11, bay looking out, car leaves | silent establishing wide | — |
- **Conditions (all through the FXI Studio connector, first frame = approved still, no end frame, 4 s, 720p, 9:16, sound on):**
  - **Economy:** P-Video.
  - **Balanced:** Grok Imagine Video 1.5.
  - **Premium reference:** Seedance 2.5 (the engine of the original series), on D and T only.
- **Replicates:** 2 per cell. Seed 101401 for replicate 1 and 101402 for replicate 2 in every condition (passed to the API; whether a model honours seeds is not verified).
- **Prompts:** one prompt per shot, identical across conditions, taken from the rebuild's shot list plus the anti-subtitle line.
- **Total:** 16 Economy/Balanced clips + 4 Premium = 20 clips. Budget cap 1,200 credits. If a dispatch would exceed the cap, trim in this order: Premium replicate 2 on T, then Premium replicate 2 on D. Any trim is reported as a deviation.

## Outcomes (automatic; code in `code/`)
1. **Word error rate (WER)**, D and T only: whisper.cpp `small.en` transcript of the clip audio vs the scripted line; lowercase, punctuation removed; word-level Levenshtein ÷ scripted word count.
2. **Burned-in text:** Tesseract OCR on 8 evenly spaced frames. A clip is text-positive if any frame has a word of ≥ 3 alphanumeric characters at confidence ≥ 70 that does not also appear in OCR of the input still. Every positive is checked by eye and the check is reported.
3. **Model-inserted cut:** ffmpeg scene score > 0.4 anywhere after frame 0.
4. **First-frame fidelity:** SSIM (grey, 360×640) between frame 0 and the input still cropped to the clip's aspect.
5. **Auto-usable:** no burned-in text, no inserted cut, and (dialogue shots) WER = 0.
6. **Cost per finished second:** (credits charged + 3 cr still) ÷ 4 s, in dollars at the Creator plan rate of $0.025 per credit. Also **cost per auto-usable second** = cost per finished second ÷ auto-usable rate of the cell.
7. Resolution, fps and duration from ffprobe.

## Human rating (pending by design)
One blinded side-by-side sheet per shot: clips labelled by letter in random order; the key is stored separately in `data/blind_key.json`. The operator rates each clip 1–5 for realism, motion, identity, lip-sync (dialogue only) and artifacts. Ratings are reported as pending until collected. No ratings will be invented.

## Analysis
Per condition and per shot: mean of each outcome with a 95% bootstrap CI (10,000 resamples, percentile). With n = 2 per cell, CIs are reported only at the condition level (n = 4 to 8); per-cell values are descriptive. A cost vs auto-usable-rate scatter marks the Pareto frontier. No p-values.

## Exclusions
A generation that fails or is refused is re-dispatched once (failed jobs are not charged) and recorded. No clip is dropped for quality. All clips, including failures, appear in `data/generations.csv`.