# MotionGPT3: Human Motion as a Second Modality

Anonymous authors
Paper Supp. Video

## MotionGPT3 is a hybrid motion-language model designed to process arbitrary input sequence of motion or language and generate outputs in either modality.

- Text-to-Motion
- Motion-to-Text

## Text-to-Motion

A person is crouched down and walking around sneakily.
A man swimming using both hands together while kicking feet in unison.
Person sits on the ledge of something then gets off and walks away.
A person stands still with both arms raised at shoulder height.
The person jumps over something and lands on feet.
A figure steps backward cockily, swinging their arms.
He is running in straight line then jumps over something and continues running.
a person walks forward with their right leg limping.
A person walks forward using left hand to steady themselves on an object.

## Motion-to-Text

"The person is walking forward and loses their balance on one foot but they continue to walk behind."
"A person walks to the right, then walks to the left, then returns to their starting position."
"A person brings the ball back to his throwing motion and throws again."
"A person takes a few steps forward, then does a cartwheel diagonally to the right then takes a step back."
"A person is shaking his hand."
"A person turns around, picks up what is moving and leans down to pick it up."
"A person holds their arms in front of them and does four squats."
"A person raises their arms around themselves and it and then motions them violently."
"A person lifts something, tilting the other person and places it back down."
"A person walks up steps, starts upstairs and then the stairs back down."
"A figure walks forward, picks something up and puts it on the table."
"A person lifts something, tilting the other person or places it back down."

## Abstract

With the rapid progress of large language models (LLMs), multimodal frameworks that unify understanding and generation have become promising, yet they face increasing complexity as the number of modalities and tasks grows. We observe that motion quantization introduces approximation errors that cap motion quality, and that unifying discrete text and continuous motion within a single-stream backbone amplifies cross-modal interference.

Motivated by recent multi-branch Transformer designs that separate signals from different modalities, we propose MotionGPT3, a bimodal motion-language model for both understanding and generation. MotionGPT3 encodes raw motion into a continuous latent space using a variational autoencoder (VAE), thereby avoiding quantization-induced artifacts, while leveraging the semantic prior of pretrained language models. A dual-stream Transformer with shared attention preserves modality-specific routes while enabling controlled, bidirectional information flow, which reduces interference, stabilizing optimization, and empirically accelerates convergence without degrading fidelity. For multimodal joint training, a generate-then-align three-stage schedule further improves stability and limits cross-task interference.

Experiments show that MotionGPT3 achieves 2× faster convergence in training loss and up to 4× faster convergence in validation, while maintaining state-of-the-art performance on standard motion understanding and motion generation benchmarks.

## Our method

We introduce a three-stage alignment for our hybrid motion-language model. First the model learn to generate motion properly. Then we further align the motion branch with language by introducing motion reasoning. Finally, we fine-tune the model by joint training with unfrozen text modules.

# Experiments

- 2× faster training convergence vs. discrete & unified baselines while maintaining or improving quality.
- State-of-the-art `R@1`/`R@3` on HumanML3D text-to-motion; lower `MMDist`↓.
- Cross-Modal Attention helps in a non-monotonic pattern, with last `L` layers enabled.
- Competitive results with a smaller motion branch (~51M params) and modest text backbones (124M).

## Training Speed

Continuous motion + bimodal architecture accelerates training by ~2× vs. discrete/unified variants while improving quality.

## Text-to-Motion Comparison

Evaluated on HumanML3D test split. Metrics: `R@1`/`R@3` (higher is better) and `MMDist`↓ (lower is better).

## Ablations on Bimodal Architecture

a) Cross-Modal Attention
CMA enabled in the last `L` layers (`L ∈ {1,…,6}`). Performance improves up to `L=5`, then slightly degrades at `L=6`, indicating a non-monotonic pattern.
b) Model Params
All models trained for 200K iterations. A 124M text branch is competitive with 355M/774M, and the motion branch achieves strong results with ~51M params (halved vs. larger variants).

# Video

## BibTeX