The reference anchor decides everything
Ten held-out clips, every model × ref-mode × task combination,
same per-entry seeds throughout.
Correction. An earlier Space reported that the audio-repair fine-tune cost ~40%
of video edit strength.
That was wrong. It compared the fine-tuned model evaluated
without the anchor against a baseline evaluated in its own native condition.
ref_prepend_warmup_steps = 0 is falsy, so train.py's annealing
branch never ran and the anchor was kept on every step — the run logged 0
[ref-warmup] lines, against the base run's 12,420.- The base checkpoint annealed the anchor away by step 5000. The two models have
opposite in-distribution conditions, and each must be read in its own.
The matrix
Video gap = mean L1(gen, target) − L1(gen, input); negative = the edit moved
toward the target instead of copying the input. Rows marked OOD
evaluate a model outside the condition it was trained in.
add starts working. It had been a pass-through failure on every previous
checkpoint — copying the input rather than synthesising the object — including across the
full 1784-clip test split. The anchor is the target's first latent frame, and for
add the target is the object-present clip, so the anchor is a frame
containing the object to be added. The base rows are the control: that anchor was always
available to the base model, which had been trained to ignore it.
Per clip
Sparse ↔ dense motion round trip
A different question on the same ten clips: how much of a real motion
field survives being squeezed through a handful of point trajectories and
regenerated? Motion-I2V's stage 1 is
the codec — a ControlNet that turns sparse drags into dense optical flow.
Read every number against the floor. The flow field is stored as a 2-channel
image through a VAE, and that VAE has a flat ~4 px reconstruction floor at
320×320 — independent of how fast anything is moving, because a 10 px
displacement is only 0.03 in the [0,1] image the VAE actually sees. Encoding and
decoding the ground truth alone, with no diffusion at all, already costs that
much. The ×floor column is the honest measure.
And against the noise. Every figure here is the mean of three seeds, with the
spread shown. That matters more than it usually does: re-running the whole pipeline
with a different seed moves the median EPE by — px
(per-seed medians —), and one clip swings 16.7→51.1 px.
A single run of this is not a result. Differences smaller than that spread —
including between this table and the single-seed version first published here
— are noise.
What survives the noise. Three things hold across every seed:
- The reconstruction sits 5–8× the VAE floor, so the sparse
bottleneck — not the decoder — is what costs the accuracy.
- Direction is recovered far better than magnitude (cos 0.71–0.82,
while the per-clip magnitude ratio spans 0.87× to 2.35×).
- Hints are followed loosely, not literally. Compare the last two columns:
the stamped hint is within ~2–4 px of the truth where it lands, but the
regenerated flow at those same pixels is 20–31 px off. The model reads a
hint as a suggestion about a region, not a constraint on a pixel.
What it does recover is
where the motion is — the moving object's
silhouette comes back sharply from 64 points.
What a drawn line can express
The clips above use 64 machine-extracted tracks. This is the other end of
the same interface — lines drawn by hand on a still photo, which is what the
Motion-I2V demo actually exposes. Each group isolates one question. Left panel is the
drawn path, the trail in the second is that path being followed frame by frame.
- direction — same start, three different lines. The hint hue, and
the field, follow the line.
- speed — same direction, 30 px vs 180 px of line: 18.6 → 53.4 px
of mean flow. Longer line, faster motion, but sub-linearly: 6× the
line gives under 3× the motion.
- path shape — a straight line and an arc to the same endpoint.
They agree at the last frame by construction; the curve shows up in the
intermediate frames, which is exactly what reference flow can represent.
- many drags — four drags radiating out, in, or tangentially read as
dolly in, dolly out and roll.
- independent — two drags in opposite directions on two birds, both
honoured at once.
- brush — the same drag with and without it: 51 px of mean flow across
the whole scene, versus 8.7 px confined to the car.
Camera motion vs object motion
Look at the hint panels above and most of what they encode is the
camera, not the subject — on these clips the global sweep is simply the
larger signal. Reference flow makes the split cheap: flow[t] already is
the frame-0→t correspondence field, so one global transform can be fitted per
frame and subtracted. What is left is the motion that is independent of the camera.
A 2-D transform cannot model parallax. Static scene content at different
depths moves differently under a translating camera, so it survives the subtraction
and shows up in the residual alongside genuine object motion — visible as
structure on the cage bars in the birds clips. The residual is "not explained by one
global transform", which is object motion plus parallax, not object motion alone.
The fit is also selected per frame from translation/affine/homography by lowest
median residual, with a dense sanity check: an unguarded homography produced a
1675 px "camera" for 234 px of real motion on one clip.