# Syncphony: Synchronized Audio-to-Video Generation with Diffusion Transformers

## Shifted Audio with the Same Image Input

Generated videos with shifted audio and the same image input demonstrate variations in motion depending on the alignment of audio cues.

##### machine gun shooting

###### Audio 1

###### Audio 2

##### striking bowling

###### Audio 1

###### Audio 2

##### playing trombone

###### Audio 1

###### Audio 2

## Comparison

Qualitative comparison of videos generated by Syncphony (Ours), AVSyncD, and Pyramid Flow (fine-tuned), which is a variant of our model without audio cross-attention layers. Our method generates motions that are temporally aligned with audio events and produces clearer motion dynamics and stable appearances, whereas AVSyncD often suffers from saturation artifacts and weakened motion.

##### frog croaking

##### lions roaring

##### machine gun shooting

##### playing cello

##### playing violin fiddle

##### dog barking

##### cap gun shooting

##### chicken crowing

##### playing trombone

##### toilet flushing

##### baby babbling crying

##### hammering

## Ablations

### 1. Motion-aware Loss

Incorporating Motion-aware Loss improves both the magnitude and temporal precision of motion, particularly at the onset and offset of dynamic actions.

##### lions roaring

##### frog croaking

### 2. Audio Sync Guidance

Applying Audio Sync Guidance captures (ASG) subtle yet important sounds and generates motion precisely aligned with the audio cues (Full Model (w=2) hits the exact target).

##### hitting with a stick

### 3. Audio RoPE

Applying Audio RoPE to the audio features shows tighter temporal alignment between motion and sound events.

##### cap gun shooting

##### playing trombone