> ## Documentation Index > Fetch the complete documentation index at: https://docs.liquid.ai/llms.txt > Use this file to discover all available pages before exploring further. # Audio Capabilities > Use LFM3-Audio for speech recognition, text-to-speech, speech translation, spoken chat, speech-to-text chat, audio understanding, and tool calling from spoken requests. LFM3-Audio is a single end-to-end speech model: it listens to audio, reads text, and answers in text, speech, or interleaved text and speech. One model covers speech recognition, text-to-speech, speech translation, spoken chat, and tool calling from spoken requests. The examples below use [LFM3-Audio-2B](https://docs.liquid.ai/lfm/models/lfm3-audio-2b) with Transformers. For images and video as well as audio, see [Omni Capabilities](https://docs.liquid.ai/lfm/key-concepts/omni-capabilities.md). ## Setup Install Transformers and the packages used by the examples. `librosa` and `soundfile` load and save audio files, and `datasets` loads a sample recording. A GPU is recommended; the model runs in bfloat16. ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} pip install "transformers>=5.20.2" accelerate datasets librosa soundfile ``` Load the model and processor with `trust_remote_code=True`. The processor bundles the audio codec, which turns generated speech codes into a 24 kHz waveform. Access it as `processor.audio_tokenizer` and move it to the model's device. Keep the codec in its own precision; don't cast it to bfloat16. ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} import torch from transformers import AutoModelForMultimodalLM, AutoProcessor model_id = "LiquidAI/LFM3-Audio-2B" device = "cuda" if torch.cuda.is_available() else "cpu" dtype = torch.bfloat16 if device == "cuda" else torch.float32 model = AutoModelForMultimodalLM.from_pretrained( model_id, dtype=dtype, device_map=device, trust_remote_code=True ).eval() processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True) codec = processor.audio_tokenizer.to(device=device).eval() ``` ### Output modes Every task is a chat conversation, and the system prompt describes the task. Text output is the default: generation starts in text mode and switches to speech or interleaved output when the model emits an output-mode token. To select a mode from the start, prefill the assistant turn: put one of the prefixes below in a final assistant message and let the model continue it. | Assistant prefill | Behavior | | - | - | | None | Start in text mode. The model can still emit a token that switches modes. | | `<\|audio_start\|>` | Start speech-only output, for example text-to-speech, until an end-of-audio frame. | | `<\|mixed_start\|>` | Start interleaved text and speech, alternating blocks of 6 text tokens and 12 speech frames. | | `` | Begin a thinking block. The answer follows the closing ``. | | `` | Supply an empty thinking block, so the answer starts immediately. | The thinking prefixes are for text-only answers. For those, also pass `suppress_mixed=True` and `suppress_audio_out=True` to `generate()`, so the model doesn't switch to speech. ### Helper The helper below runs one assistant turn and returns the text and the speech waveform, if any. When you pass `prefill`, it sets `continue_final_message=True`, so generation continues the prefilled message instead of starting a new assistant turn. For speech and interleaved output, pass `SPEECH_SAMPLING`. These values are also the defaults in the model's `generation_config.json`; passing them explicitly keeps the examples reproducible. ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} import soundfile as sf SAMPLE_RATE_OUT = 24_000 # the codec decodes to 24 kHz SPEECH_SAMPLING = dict(audio_temperature=1.0, audio_top_k=128, audio_repetition_penalty=1.1, audio_repetition_window=4096) @torch.inference_mode() def generate(conversation, prefill=None, max_new_tokens=512, **generate_kwargs): """Runs one assistant turn. Returns (text, waveform or None).""" if prefill is not None: conversation = conversation + [{"role": "assistant", "content": prefill}] inputs = processor.apply_chat_template( conversation, add_generation_prompt=prefill is None, continue_final_message=prefill is not None, tokenize=True, return_dict=True, return_tensors="pt", ).to(device) outputs = model.generate(**inputs, max_new_tokens=max_new_tokens, **generate_kwargs) text = processor.decode(outputs.sequences[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True) waveform = None if outputs.audio_codes is not None and outputs.audio_lengths[0] > 0: codes = outputs.audio_codes[:1, :, : outputs.audio_lengths[0]] waveform = codec.decode(codes).audio_values[0].float().cpu().numpy().squeeze() return text, waveform ``` ## Speech recognition Use the system prompt `Perform ASR in .` with the English name of the language, or `Perform ASR.` to let the model detect it. Pass audio in the user turn as `{"type": "audio", "audio": ...}`, where the value is a file path or URL. The processor resamples it to 16 kHz mono. This example transcribes a LibriSpeech clip: ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} import io from datasets import Audio as AudioFeature, load_dataset librispeech = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") librispeech = librispeech.cast_column("audio", AudioFeature(decode=False)) sample = librispeech[0] speech, sr = sf.read(io.BytesIO(sample["audio"]["bytes"])) sf.write("librispeech.wav", speech, sr) print("Reference:", sample["text"]) transcript, _ = generate([ {"role": "system", "content": "Perform ASR in English."}, {"role": "user", "content": [{"type": "audio", "audio": "librispeech.wav"}]}, ]) print("Transcript:", transcript) ``` **Output** ```text theme={"theme":{"light":"github-light","dark":"github-dark"}} Reference: MISTER QUILTER IS THE APOSTLE OF THE MIDDLE CLASSES AND WE ARE GLAD TO WELCOME HIS GOSPEL Transcript: Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel. ``` ## Live speech recognition Live speech recognition transcribes speech while it is still being spoken, instead of waiting for the full recording. Use the system prompt `Perform live ASR.`. It currently supports English only. The model processes audio in chunks of about 0.96 s and writes the text for each chunk as soon as the chunk arrives. `model.transcribe_live()` runs one chunk at a time and returns a state that you pass to the next call. In an application, you open a microphone stream into the model and a text stream out of it. This example simulates the microphone by feeding the LibriSpeech clip in 100 ms pieces: ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} import librosa tokenizer = processor.tokenizer chunk_frames = 8 * model.config.interleaved_n_audio prompt = tokenizer( "<|startoftext|><|im_start|>system\nPerform live ASR.<|im_end|>\n<|im_start|>assistant\n", add_special_tokens=False, return_tensors="pt", ).input_ids.to(device) suppress = tokenizer.convert_tokens_to_ids(["<|mixed_end|>", "<|endoftext|>"]) stop = tokenizer.convert_tokens_to_ids(["<|endoftext|>", "<|im_start|>"]) state, feature_cache = None, None mel = torch.empty((1, processor.feature_extractor.feature_size, 0), device=device) def transcribe(frames, is_final=False): global state output = model.transcribe_live( frames, state, input_ids=prompt if state is None else None, is_final=is_final, max_new_tokens=96 if is_final else 48, suppress_token_ids=suppress, stop_token_ids=stop, ) state = output.state print(tokenizer.decode(output.sequences[0], skip_special_tokens=True), end="", flush=True) waveform, _ = librosa.load("librispeech.wav", sr=16_000) for start in range(0, len(waveform), 1_600): # 100 ms microphone chunks features, feature_cache = processor.feature_extractor.stream( waveform[start:start + 1_600], feature_cache ) mel = torch.cat((mel, features.input_features.to(device)), dim=-1) while mel.shape[-1] >= chunk_frames: transcribe(mel[..., :chunk_frames]) mel = mel[..., chunk_frames:] # Flush the remaining audio and end the transcript features, feature_cache = processor.feature_extractor.stream([], feature_cache, is_final=True) mel = torch.cat((mel, features.input_features.to(device)), dim=-1) while mel.shape[-1] > chunk_frames: transcribe(mel[..., :chunk_frames]) mel = mel[..., chunk_frames:] transcribe(mel, is_final=True) print() ``` **Output** ```text theme={"theme":{"light":"github-light","dark":"github-dark"}} Mr. Quilter is the apostle of the middle classes, and we're glad to welcome his gospel. ``` For streams longer than 30 seconds, pass `input_ids=prompt` again to start a new text context. The encoder keeps following the audio stream. ## Text-to-speech Use the system prompt `Perform TTS in using a voice.` and prefill `<|audio_start|>`. Supported languages and accents: Arabic, Danish, German, Greek, British English, American English, Finnish, French, Hebrew, Italian, Japanese, Korean, Dutch, Polish, Portuguese, Spanish, Swedish, Turkish, and Chinese. ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} torch.manual_seed(0) _, tts_en = generate( [ {"role": "system", "content": "Perform TTS in American English using a female voice."}, {"role": "user", "content": "I made a little plan for today: go for a walk, find a new workout playlist, and try a new cake recipe"}, ], prefill="<|audio_start|>", **SPEECH_SAMPLING, ) sf.write("tts_en.wav", tts_en, SAMPLE_RATE_OUT) _, tts_fr = generate( [ {"role": "system", "content": "Perform TTS in French using a male voice."}, {"role": "user", "content": "On pourrait inventer une nouvelle recette ce soir. Si le résultat est étrange, on lui donnera un nom très chic."}, ], prefill="<|audio_start|>", **SPEECH_SAMPLING, ) sf.write("tts_fr.wav", tts_fr, SAMPLE_RATE_OUT) ``` As a quick check, let the model transcribe its own French speech: ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} transcript, _ = generate([ {"role": "system", "content": "Perform ASR in French."}, {"role": "user", "content": [{"type": "audio", "audio": "tts_fr.wav"}]}, ]) print("Transcript:", transcript) ``` **Output** ```text theme={"theme":{"light":"github-light","dark":"github-dark"}} Transcript: On pourrait inventer une nouvelle recette ce soir. Si le résultat est étrange, on lui donnera un nom très chic. ``` ## Speech translation LFM3-Audio translates speech into text. Translation works best into or out of English. Name the source and target languages in the system prompt, for example `Convert the speech in {source} to written {target}.` or `Translate the following speech from {source} to {target} in text form.` The model was trained on many phrasings, so pick the one that works best for your data. For text-to-text translation, use the text model [LFM3-1.7B](https://docs.liquid.ai/lfm/models/lfm3-1.7b) instead. This example translates the French clip from the text-to-speech example into English: ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} translation, _ = generate([ {"role": "system", "content": "Convert the speech in French to written English."}, {"role": "user", "content": [{"type": "audio", "audio": "tts_fr.wav"}]}, ]) print("Translation:", translation) ``` **Output** ```text theme={"theme":{"light":"github-light","dark":"github-dark"}} Translation: We could invent a new recipe this evening. If the result is strange, it will be given a very fancy name. ``` ## Spoken chat With the system prompt `Respond with interleaved text and audio.` and the `<|mixed_start|>` prefill, the model answers conversationally in text and speech at the same time, so speech can start playing before the full answer is written. Keep the system prompt exactly as shown, and give any further instructions in the user turn. This example first records a spoken question with text-to-speech, then asks it: ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} _, question = generate( [ {"role": "system", "content": "Perform TTS in American English using a male voice."}, {"role": "user", "content": "I want to move more, but I have not found an activity I enjoy. How can I try a few options and build a small habit?"}, ], prefill="<|audio_start|>", **SPEECH_SAMPLING, ) sf.write("question.wav", question, SAMPLE_RATE_OUT) reply, reply_speech = generate( [ {"role": "system", "content": "Respond with interleaved text and audio."}, {"role": "user", "content": [{"type": "audio", "audio": "question.wav"}]}, ], prefill="<|mixed_start|>", max_new_tokens=1024, **SPEECH_SAMPLING, ) print("Text:", reply) sf.write("reply.wav", reply_speech, SAMPLE_RATE_OUT) ``` **Output** (text part; the speech is saved to `reply.wav`) ```text theme={"theme":{"light":"github-light","dark":"github-dark"}} Text: Try a short trial of each activity, like a weekend hike, a cooking class, or a volunteer shift, and note how much fun you had each time. Pick the one that feels most appealing and commit to doing it for just one week to see if it sticks. Does a one‑week trial of a single activity sound doable? ``` For streaming applications, `model.generate()` accepts `streamer` for text tokens and `audio_streamer` for speech frames, and the codec decodes those frames incrementally, so speech can be played as it is generated. ## Speech-to-text chat In this mode, the model behaves like a text LLM that also accepts speech. No system prompt is needed. Prefill `` to let the model reason first; it closes the reasoning with ``, and the answer follows. Pass `suppress_mixed=True` and `suppress_audio_out=True` to keep the answer in text. ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} text, _ = generate( [{"role": "user", "content": [{"type": "audio", "audio": "question.wav"}]}], prefill="", max_new_tokens=2048, suppress_mixed=True, suppress_audio_out=True, ) thinking, _, answer = text.partition("") print("Thinking:", thinking.strip()) print("Answer:", answer.strip()) ``` **Output** (truncated) ```text theme={"theme":{"light":"github-light","dark":"github-dark"}} Thinking: We need to answer the question: "I want to move more, but I have not found an activity I enjoy. How can I try a few options and build a small habit?" 1. **Analyze the User's Request:** * **Goal:** Move more. * **Problem:** Hasn't found an activity they enjoy. * **Solution Needed:** How to try a few options and build a small habit. ... Answer: It is completely normal to struggle to find an activity you enjoy when you want to move more. The key to overcoming this is to break the process into tiny, manageable steps and focus on the *process* rather than the *result*. ... ``` To skip thinking, prefill an empty `` block, so the answer starts immediately: ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} answer, _ = generate( [{"role": "user", "content": [{"type": "audio", "audio": "question.wav"}]}], prefill="", max_new_tokens=1024, suppress_mixed=True, suppress_audio_out=True, ) print("Answer:", answer.strip()) ``` ## Audio understanding Text and audio can be mixed in the same user turn, for example a recording plus a written question about it. Questions can address speech, music, background sounds, or events. ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} answer, _ = generate( [{"role": "user", "content": [ {"type": "audio", "audio": "librispeech.wav"}, {"type": "text", "text": "Who or what is the speaker talking about, and what is their attitude towards it?"}, ]}], prefill="", max_new_tokens=512, suppress_mixed=True, suppress_audio_out=True, ) print("Answer:", answer.strip()) ``` **Output** ```text theme={"theme":{"light":"github-light","dark":"github-dark"}} Answer: The speaker is talking about **Mr. Quilter**, who is described as "the apostle of the middle classes." The speaker's attitude is one of welcome and approval—"we're glad to welcome his gospel," indicating a positive, supportive stance toward Mr. Quilter's message. ``` ## Tool calling This example turns one spoken request into two function calls: cancel a streaming-platform subscription at the end of the billing period, then subscribe to a podcast platform's Max plan with monthly billing. StreamBox and PodSpace are fictional services, and the example only prints the proposed calls. List the tools in the system prompt, between `` and ``, one JSON object per line. The user asks by voice, and the model returns one `` block per call. ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} import json tools = [ { "name": "cancel_subscription", "description": "Cancel the signed-in user's subscription to a streaming platform at the specified time.", "parameters": { "type": "object", "properties": { "platform": {"type": "string", "description": "Name of the streaming platform, for example StreamBox."}, "effective_when": { "type": "string", "enum": ["immediately", "end_of_billing_period"], }, }, "required": ["platform", "effective_when"], }, }, { "name": "subscribe_to_podcast_platform", "description": "Subscribe the signed-in user to a podcast platform with the chosen plan and billing frequency.", "parameters": { "type": "object", "properties": { "platform": {"type": "string", "description": "Name of the podcast platform, for example PodSpace."}, "plan": {"type": "string", "enum": ["basic", "plus", "max"]}, "billing_frequency": {"type": "string", "enum": ["monthly", "annual"]}, }, "required": ["platform", "plan", "billing_frequency"], }, }, ] tool_prompt = ( "Return tool calls for both requested actions, in the order requested. " "Do not claim that an action has completed without a tool result.\n" "List of tools:\n\n" + "\n".join(json.dumps(t) for t in tools) + "\n" ) # Record the spoken request _, request = generate( [ {"role": "system", "content": "Perform TTS in British English using a female voice."}, {"role": "user", "content": "Please cancel my StreamBox subscription starting December first. Then subscribe me to PodSpace on the Max plan, billed monthly."}, ], prefill="<|audio_start|>", **SPEECH_SAMPLING, ) sf.write("request.wav", request, SAMPLE_RATE_OUT) call, _ = generate( [ {"role": "system", "content": tool_prompt}, {"role": "user", "content": [{"type": "audio", "audio": "request.wav"}]}, ], max_new_tokens=512, suppress_mixed=True, suppress_audio_out=True, ) print(call) ``` **Output** ```text theme={"theme":{"light":"github-light","dark":"github-dark"}} StreamBox end_of_billing_period PodSpace max monthly ``` Generated calls use the LFM3 `` XML format. Your application parses and executes the calls and returns the results as `tool` messages. See [Tool Use](https://docs.liquid.ai/lfm/key-concepts/tool-use.md) for the full workflow. Don't prefill a thinking block for tool calls; the model calls tools without reasoning first. To get a spoken answer once the tool results are in, start the system prompt with `Respond with function calls, and interleaved text and audio.\n\n`, followed by any instructions and the tool list, and don't pass `suppress_mixed` or `suppress_audio_out`. The model then either calls tools or answers in interleaved text and speech. The model card's [tool calling example](https://huggingface.co/LiquidAI/LFM3-Audio-2B) shows the full loop, including returning tool results. ## FAQ **Which model should I use?** Use `LFM3-Audio-2B` for speech and audio tasks: transcription, speech synthesis, translation, spoken chat, audio understanding, and spoken tool calls. Use `LFM3-Omni-2B` when the request also involves images or video. See [Omni Capabilities](https://docs.liquid.ai/lfm/key-concepts/omni-capabilities.md). **What audio input is supported?** Audio files are resampled to 16 kHz mono on load. Speech output is a 24 kHz waveform. **Can the model stream speech?** Yes. LFM3-Audio-2B uses a causal audio encoder that supports live audio input (see [Live speech recognition](#live-speech-recognition)), and its audio codec supports streaming decoding. In Transformers, pass `streamer` and `audio_streamer` to `model.generate()`. **Does the model reason before answering?** For text-only answers, prefill `` to let the model reason first, or `` to answer directly. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.