# SpeechOp: Inference-Time Task Composition for Generative Speech Processing

## SpeechOp Demo

### Abstract

While generative Text-to-Speech (TTS) systems leverage vast "in-the-wild" data to achieve remarkable success, speech-to-speech processing tasks like enhancement face data limitations, which lead data-hungry generative approaches to distort speech content and speaker identity. To bridge this gap, we present SpeechOp, a multi-task latent diffusion model that transforms pre-trained TTS models into a universal speech processor capable of performing a wide range of speech tasks and composing them in novel ways at inference time. By adapting a pre-trained TTS model, SpeechOp inherits a rich understanding of natural speech, accelerating training and improving S2S task quality, while simultaneously enhancing core TTS performance. Finally, we introduce Implicit Task Composition (ITC), a novel pipeline where ASR-derived transcripts (e.g., from Whisper) guide SpeechOp's enhancement via our principled inference-time task composition. ITC achieves state-of-the-art content preservation by robustly combining web-scale speech understanding with SpeechOp's generative capabilities.

## Speech Processing Example 1: obj1_noisy

Noisy Input
SpeechOp Enhancement
SpeechOp Enhancement with TTS Guidance
SpeechOp Foreground Separation
SpeechOp Background Separation

## Speech Processing Example 2: obj0_noisy

Noisy Input
SpeechOp Enhancement
SpeechOp Enhancement with TTS Guidance
SpeechOp Foreground Separation
SpeechOp Background Separation

## Speech Processing Example 3: Reverberant Speech

Clean Reference (DAPS)
Reverberant Input (DAPS)
SpeechOp Enhancement
SpeechOp ITC Guided Enhancement
SpeechOp Personalized Enhancement
SpeechOp Foreground Separation
SpeechOp Foreground Separation with TTS Guidance

## LibriMix: Speaker Separation

### Clean Background

Mixture
Target Speaker 0
SpeechOp Separation 0
SpeechOp Separation 0 with TTS Guidance
Target Speaker 1
SpeechOp Separation 1
SpeechOp Separation 1 with TTS Guidance

### Noisy Background

Mixture
Target Speaker 0
SpeechOp Separation 0
SpeechOp Separation 0 with TTS Guidance
Target Speaker 1
SpeechOp Separation 1
SpeechOp Separation 1 with TTS Guidance

## Zero-Shot TTS Examples

### Example 1

Original Prompt
SpeechOp Result

### Example 2

Original Prompt
SpeechOp Result

## Real Edit Examples

### Example 1

Original Audio
SpeechOp Edit

### Example 2

Original Audio
SpeechOp Edit