Skip to main content
Speech-capable models can answer in text, speech, or interleaved text and speech. This guide covers text-to-speech, spoken chat, and spoken answers about images. Use an audio model for speech-only tasks. Use an omni model to answer questions about a photo in speech. The examples generate speech as a 24 kHz waveform. For speech input, see Audio Capabilities.

Setup

Load the model and processor as shown in Audio Capabilities. Where the code differs between audio and omni models, pick a model type in the tabs. The choice applies to every example on the page. Speech output uses each model’s recommended sampling parameters. They are also the defaults in the model’s generation_config.json; the examples pass them explicitly to keep the results reproducible. Check the model page for the values.

Choose the output modality

Generation starts in text mode and switches to speech or interleaved output when the model emits an output-mode token. To select a mode from the start, prefill the assistant turn: add an assistant message with one of the prefixes below and pass continue_final_message=True to apply_chat_template().The thinking prefixes are for text-only answers. For those, also pass suppress_mixed=True and suppress_audio_out=True to model.generate(), so the model doesn’t switch to speech.

Text-to-speech

Use the system prompt Perform TTS in <language> using a <female|male> voice. For English, say American English or British English. Check the model page for supported languages and voices. This example speaks a sentence in American English:
Prefill <|audio_start|>:
To check the result, pass the file to the speech recognition example.

Spoken chat

Use the system prompt Respond with interleaved text and audio. The model answers conversationally in text and speech at the same time, so speech can start playing before the full answer is written. Keep the system prompt exactly as shown, and give any further instructions in the user turn. This example first records a spoken question with text-to-speech, then asks it and saves the spoken reply to reply.wav:
Record the question:
Then ask it. Prefill <|mixed_start|>:
Output (text part; the speech is saved to reply.wav)

Answer in speech

Spoken answers about images require an omni model with image input and speech output. Load it as shown in Audio Capabilities.
An omni model can answer a question about an image in speech. Add an output instruction after the task prompt, for example Respond with interleaved text and audio. This example asks in text about a sample photo:
Output (text part; the speech is saved to bee_answer.wav)

Stream speech

For streaming applications, model.generate() accepts streamer for text tokens and audio_streamer for speech frames. The codec decodes those frames incrementally, so speech can be played as it is generated.

FAQ

Which model should I use? Choose an audio model for speech-only workloads or an omni model when the spoken answer is about an image or a video. See the model library for current options. What audio format is the speech output? A 24 kHz mono waveform. The examples save it as WAV with soundfile. Does the model reason before speaking? The examples on this page do not use thinking for speech output. Check Thinking for reasoning support and controls when requesting a text answer.