Real-time Interactive Streaming Demo on a single H100 GPU: Using simple drag and mouse motions, MotionStream
enables real-time control of diverse scenarios including both object motion and camera movement with
various grid configurations. Network latency plays an important role in streaming performance (some
demo samples may exhibit slightly higher latency due to intercontinental connections). Our method
maintains stable streaming with constant attention context. The system demonstrates robust
long-video generation capabilities: by anchoring to the clean first chunk with a single attention
window (keeping minimal drifted chunks within the attention context), the model often recovers
quality even after disruptions, showcasing its resilience in long video streaming scenarios.
Notation: C = chunk size, S = number of sink
chunks, W = local window size. Results demonstrate that maintaining at least one
sink chunk is crucial for stable long-video generation. All experiments use our 1.3B causal
distilled model, showing the effectiveness of our attention sink mechanism for extended temporal
extrapolation.
Sample 1: Extrapolation Comparison
Sample 2: Extrapolation Comparison
Comparison with Other Baselines: We compare our methods with recent
motion-controlled video generation approaches using samples from Sora demos. We denote our models as
1.3B-T (1.3B parameter teacher model), 1.3B-S (1.3B parameter
student/distilled model), 5B-T (5B parameter teacher model), and
5B-S (5B parameter student/distilled model). Our methods consistently demonstrate
high video quality and motion adherence compared to baseline approaches, with the student models
achieving real-time performance.
Motion Transfer: Given an initial image of similar structure, MotionStream can naturally transfer motions of arbitrarily long videos (can be done in real-time with online trackers).
Guidance Ablation Study: Using high motion guidance leads to overly rigid translations due to strict track adherence and ignores text cues (below sample is prompted with "rainbow appearing at the back"). While prompt guidance alone shows inferior quantitative metrics for motion reconstruction, it enables flexible text-based control. Our proposed joint guidance strategy effectively balances these two.