Supplementary material · Anonymous submission

Sound Sparks Motion

Audio and Text Tuning for Video Motion Editing

A training-free framework for motion editing in audio-visual video generation models. Audio is a handle for motion — the optimized latent is discarded after editing.

Scroll for videos and interactive results

Videos

The full supplementary video and the baseline comparison reel.

Supplementary video Method, qualitative results, transfer, and generalization to H3 · 6:05
Baseline comparison 12 scenarios against 7 baselines, plus the H3 study · 6:40

Results on LTX-2

Hit Spark the motion! on any card to apply its edit.

Generalization to H3

Percentages are from a human study of 25 scenarios with 15 participants (375 judgements): on each card, how often ours was preferred, judged more natural, and judged the stronger edit, against the same for H3 alone. Ties make up the rest. These clips carry sound.

Audio, text, or both

The audio latent alone, the text residual alone, and the two together, beside H3 with no editing. The reel opens with the human study over all 12 scenarios (15 participants), then shows the 5 where the combination won clearly.

Audio, text, or both The two tuned signals ablated on H3, with the human study · 1:57

Baseline comparison

Twelve scenarios against seven baselines. Pick a scenario to compare. In a study of 21 participants, our result was ranked first in 56.3% of trials, against 21.4% for the next best method.

Motion transfer

The audio latent and text residual tuned on one clip, applied to another. Hit Transfer the motion! to carry them across.

The boy is laughing out loud.
Optimized onA man laughing out loud
audio latenttext residual
New input
A cat yawning.
Optimized onA dog yawning
audio latenttext residual
New input
A man shouting.
Optimized onAn angry man shouting
audio latenttext residual
New input
A red bird opening its wings.
Optimized onA bird opening its wings
audio latenttext residual
New input

Where each signal attends

Average cross-attention for the two tuned signals, over the first frame of the result. A2V is audio-to-video, T2V is text-to-video; switch between them on any card.

"A bird opening its wing."
"Turtle extending its neck."
"A goldfish jumps out of the fish tank into the air."
"The man climbs up the rock."
"A man is laughing out loud."
"The man pets the dog."
"The robot waving its hand."
"A dog yawning."

Ablations

Every variant of a scenario plays at once. Each reel replays its first scenario at half speed.

Effect of the supervision signal CLIP vs. XCLIP vs. a vision-language model, on 4 scenarios · 1:31
Effect of the regularizers None, LPIPS, temporal, latent, and all three, on 4 scenarios · 1:26
Rebuttal

Rebuttal Results

Additional results referenced in our responses to the reviewers. All clips play muted; use the player controls to turn sound on. Each card uses the input clip’s soundtrack.

IVEBench: subject-motion edits

Edits from IVEBench’s subject-motion category, made with our method and with prompt-only LTX-2 Retake. The plot shows the official IVEBench metrics; each card shows the input and our result for the instruction above it.

Mean over items for our method, prompt-only Retake and the unedited input. The highlighted axes are the Compliance dimension, which measures whether the instructed motion appears.

Input scores 1.0 on semantic fidelity by construction, since it is compared with itself.

“Make the bird hop around instead of walking and foraging”
0:00
Input
Ours
“Make all people in the brainstorming session move to the left”
0:00
Input
Ours
“Change the woman's jogging to taking off.”
0:00
Input
Ours
“Make the person on the cliff fall down.”
0:00
Input
Ours
“Make the couple start spinning while embracing”
0:00
Input
Ours
“Show the drone taking off vertically from the landing pad”
0:00
Input
Ours
“The animals start fighting”
0:00
Input
Ours
“Make the pigeons fly.”
0:00
Input
Ours
“Make the soldier turn around on the ship”
0:00
Input
Ours
“Make the woman on the yacht wave her hand”
0:00
Input
Ours
“make the characters fly”
0:00
Input
Ours
“Make the woman on the surfboard perform a quick turn”
0:00
Input
Ours
“Make the person holding the soccer ball run instead of walking”
0:00
Input
Ours
“Make the woman stand up.”
0:00
Input
Ours
“Make the man in the black leather jacket stand up and stretch”
0:00
Input
Ours
“Make the woman tilt her head to the right”
0:00
Input
Ours
“Make the man in the suit start walking forward”
0:00
Input
Ours

Best-of-8 vs. ours

Prompt-only LTX-2 Retake rendered with 8 seeds per scenario; the sample that the same critic scores highest is shown next to our result. The note under the best-of-8 column describes what that video shows.

“The car front lights turn on and off periodically.”
0:00
Input
Ours
Best-of-8Lights switch on once and stay on.
“A bird openning its wings.”
0:00
Input
Ours
Best-of-8Body and wings smear into blur.
“A goldfish jumps out of the fish tank into the air.”
0:00
Input
Ours
Best-of-8A fish appears in the tank instead of one jumping out.
“Monkey reaching for a fruit.”
0:00
Input
Ours
Best-of-8A fruit appears in the monkey's hand.
“The man climbs up the rocks and looks at the horizon from the top.”
0:00
Input
Ours
Best-of-8Abrupt, exaggerated movement.
“A red rose blooming.”
0:00
Input
Ours
Best-of-8The rose stays closed.
“The robot waving its hand.”
0:00
Input
Ours
Best-of-8Waving hand blurs and distorts.
“Turtle extending its neck.”
0:00
Input
Ours
Best-of-8Neck becomes a thin stalk.

Noise optimization vs. ours

Gradient-based optimization of the initial noise of the Retake segment, with the same critic and the audio and text conditioning frozen, shown for each learning rate tried. The default is the learning rate with the highest critic score in its own optimization (marked with a dot); ours is the result from the paper.

“The man pets the dog.”
0:00
Input
Noise optimizationlr 0.005Best by the critic’s own score
Ours
Learning rate
“A red car door opens.”
0:00
Input
Noise optimizationlr 0.005Best by the critic’s own score
Ours
Learning rate
“Turtle extending its neck.”
0:00
Input
Noise optimizationlr 0.02Best by the critic’s own score
Ours
Learning rate
“A dog yawning”
0:00
Input
Noise optimizationlr 0.0025Best by the critic’s own score
Ours
Learning rate

Motion transfer

An audio latent and text residual optimized on one clip, applied unchanged to a different video with no further tuning on it. Hit Transfer the motion! to switch from the new input to the transferred result.

A lion roaring.
Optimized onAn angry man shouting
audio latenttext residual
New input
A man is laughing out loud.
Optimized onA man laughing out loud
audio latenttext residual
New input
An old man is laughing out loud.
Optimized onA man laughing out loud
audio latenttext residual
New input
An old man yawning.
Optimized onA dog yawning
audio latenttext residual
New input
The boy is laughing out loud.
Optimized onA man laughing out loud
audio latenttext residual
New input
A man is laughing out loud.
Optimized onA man laughing out loud
audio latenttext residual
New input
The man gently strokes the woman's hair.
Optimized onThe man pets the dog
audio latenttext residual
New input
A monkey opening its arms wide.
Optimized onA bird opening its wings
audio latenttext residual
New input
A cat yawning.
Optimized onA dog yawning
audio latenttext residual
New input
A duck opening its wings.
Optimized onA bird opening its wings
audio latenttext residual
New input
An eagle opening its wings.
Optimized onA bird opening its wings
audio latenttext residual
New input
A man is laughing out loud.
Optimized onA man laughing out loud
audio latenttext residual
New input
A lion yawning.
Optimized onA dog yawning
audio latenttext residual
New input
A child is laughing out loud.
Optimized onA man laughing out loud
audio latenttext residual
New input
An eagle opening its wings.
Optimized onA bird opening its wings
audio latenttext residual
New input
A koala yawning.
Optimized onA dog yawning
audio latenttext residual
New input
A man pets the dog.
Optimized onThe man pets the dog
audio latenttext residual
New input