Keeping One Character Across AI Video Cuts (And the Bug That Fails Silently)
A first-hand workflow for holding one character across separate AI-generated video cuts: three-view chroma-key reference sheets, scene-only prompts, and the silently-ignored reference input that makes the whole thing look like it is working when it is not.
TL;DR — To keep one character across separate AI video cuts, generate a three-view character sheet on a flat chroma-key background, pass all three images as references, and describe only the scene in the prompt. The single hardest part is not the art direction. It is that the reference input fails silently if you pass it in the wrong shape, and a plausible-looking result comes out either way.
Character consistency is the thing that separates a demo clip from a sequence you can cut together. Two videos, three cuts each, generated on one consumer GPU, are the working example throughout — and the writeup includes the mistake that cost a full day, because that mistake is invisible by design.
The pipeline
Three steps, and the third one is the one people skip.
| Step | What | Why |
|---|---|---|
| 1 | Flux generates three character sheets on flat green — front face, side face, full body. Same seed. | The identity has to exist as pixels before it can be referenced |
| 2 | All three go into the reference-to-video node as separate reference slots | One view carries only the information in that view |
| 3 | The video prompt describes the scene and action only — no appearance | Anything you describe overrides the reference |
Roughly five minutes per cut at 768×1344, 124 frames, 8 steps. A 15-second piece is three cuts, so about fifteen minutes of compute.
The prompt for those three cuts never mentioned red hair, amber eyes, or the bodysuit. It described a ruined street, a sprint, and a leap. Everything recognisable about the character came from the sheet.
Cut 1 — ignition
Cut 2 — sprint
Cut 3 — leap
Those are three separate generations, not one clip cut into three. Nothing was carried between them except the same three reference images.
The trap: the reference input fails silently
This is worth its own section because it produces no error, no warning, and a perfectly reasonable-looking video.
The node takes multiple reference images. The intuitive way to pass them is a nested object:
# WRONG — silently ignored
"ref_images": {"ref_image_0": ["40", 0]}
# RIGHT — flat dot notation, as in the official workflow
"ref_images.ref_image_0": ["40", 0]
The execution engine resolves a value into an image link only when that value is literally [node_id, output_index] at the position it inspects. It does not walk into nested dictionaries. So the inner ["40", 0] is never substituted, the node runs with zero references, and nothing complains.
Three things make this hard to catch:
The output still looks right. If your prompt describes the character's appearance — which is the natural thing to do — you get that character with zero references attached. The result is convincing, so you conclude the reference is working.
The error message points the wrong way. Passing ref_image_0 at the top level produces unexpected keyword argument 'ref_image_0'. Did you mean 'ref_images'?, which reads as an instruction to nest it. Reading the node source reinforces the misreading: it iterates (ref_images or {}).values(). Both signals point at the wrong answer, and the right answer is a third form neither suggests.
The fix came from the workflow file, not the code. The shipped template JSON declares the input name as "name": "ref_images.ref_image_0". The general lesson is worth keeping: when a node's API is ambiguous, read the official working workflow before reading the source. The source tells you what the function accepts; the workflow tells you what the engine actually sends.
How to prove the reference is doing anything
Since a wrong result and a right result look similar, the test has to remove the confound. Empty the appearance out of the prompt entirely and run the comparison:
| Condition | Prompt | References | Result |
|---|---|---|---|
| A | Scene only | 3 images | Matches the character sheet |
| B | Scene only | none | An arbitrary person |
If A and B look the same, the references are being ignored regardless of what your code appears to say. There is a second, cruder signal: reference processing adds roughly a minute per cut, so a suspiciously fast run is a hint.
Why a flat green background
A reference image transfers everything in it, not just the subject. A character sheet shot against a sky or a field drags that setting into every cut that uses it, which defeats the point of writing a different scene each time. On flat green, the only thing left to transfer is the person — lighting, environment and mood stay under the prompt's control.
Keep the sheet lighting neutral for the same reason. A warm key light in the reference tints the character in every subsequent cut.
Why three views and not one
Each view carries different information, and the missing one shows up as a specific failure:
| View | Carries | Failure without it |
|---|---|---|
| Front face | Features, eye colour | Identity drifts between cuts |
| Side face | Nose and jaw line, back of head | Profile shots get an invented face |
| Full body | Leg length, shoulder-to-waist ratio, lower costume | Proportions collapse |
The third one is the least obvious and the most damaging. A model that has only seen a face will guess at the body, and the guess changes shot to shot.
One practical note on generating the sheet: asking the image model for an extreme facial close-up mostly does not work — repeated attempts still framed the upper body. Detecting the subject against the green background and cropping the head region is more reliable than arguing with the prompt.
The rule that explains most failures
What the prompt states wins. The reference only fills what the prompt leaves empty.
Three separate failures traced back to this single rule:
- An East Asian reference produced a Western-looking character — because the prompt said nothing about ethnicity, so the model used its own default rather than the reference.
- Swapping in a new character reference kept producing the old character — because the costume description was still sitting in the prompt.
- The character stayed consistent even with references disabled — because the prompt described the full appearance, masking the bug above.
The practical consequence: if you change the reference, you must change the prompt's description of the person too — or, better, remove appearance from the prompt entirely and let the reference own it. The second option also makes the reference's contribution measurable, which is what makes the ablation test above possible.
A second character run through the identical process, with a different sheet and different scenes:
Cut 1 — shallows
Cut 2 — the falls
Cut 3 — close-up
This set is the more demanding test of the two. The three shots sit at very different distances — a full-body wide, a figure small in a landscape, and a face filling the frame — and each was generated on its own. A method that only holds at one shot scale is not much use for cutting a sequence together. Audio in each clip is generated by the video model alongside the picture, not added afterwards.
Where it still slips
Being specific about the limits is more useful than claiming it just works.
The second cut of the first sequence came out in daylight, while the first and third cuts are at night. That was not a model failure — the second cut's prompt was the only one that never mentioned time of day. The same rule that makes the method work in the first place applies here in reverse: an unstated variable is not preserved, it is re-decided. Continuity of setting has to be written into every cut, exactly like continuity of appearance would if you were not using references.
Composition needs stating too. Without an explicit framing instruction the camera pulls back, especially with more than one subject in frame, and a face that occupies a small share of a heavily downsampled latent grid has almost no tokens to be drawn with. Putting medium shot or close-up in every cut is not decoration.
Environment
Measured on a single RTX 5080 (16GB), ComfyUI 0.33.1, torch 2.12.0+cu130, 31GB system RAM. Reference-to-video model minimax_h3_ref2va_pruned_int8_convrot with the reference-to-video turbo LoRA. The LoRA ships as a 4-step variant, but 4 steps break down on large motion — 8 steps holds together and is what every number here was produced at.
FAQ
Why does the reference image get ignored without an error?
The execution engine only treats a value as an image link when it is exactly [node_id, output_index] at the level it inspects, and it does not descend into nested dictionaries. A nested {"ref_image_0": [...]} is therefore never resolved, the node runs with no references, and no exception is raised. Use the flat "ref_images.ref_image_0" key instead.
Should the character sheet have a background?
No. A reference transfers everything in the image, including the setting. A flat chroma-key background leaves only the character, so each cut's prompt can set its own environment. Keep the sheet's lighting neutral as well, since a colour cast in the reference carries into every cut.
Can I describe the character in the video prompt as well, just to be safe?
It is counterproductive. Anything stated in the prompt overrides the reference, so describing appearance both weakens the reference's contribution and hides whether it is working at all. Describe the scene and the action, and let the reference carry the identity.
How long does a 15-second sequence take?
About fifteen minutes of generation on the hardware above — three cuts of 5.2 seconds each, roughly five minutes per cut at 768×1344, 124 frames, 8 steps. That excludes the character sheet, which is a few minutes more.
← Back to all posts