Audio Capabilities

LFM3-Audio is a single end-to-end speech model: it listens to audio, reads text, and answers in text, speech, or interleaved text and speech. One model covers speech recognition, text-to-speech, speech translation, spoken chat, and tool calling from spoken requests. The examples below use LFM3-Audio-2B with Transformers. For images and video as well as audio, see Omni Capabilities.

Setup

Install Transformers and the packages used by the examples. librosa and soundfile load and save audio files, and datasets loads a sample recording. A GPU is recommended; the model runs in bfloat16.

pip install "transformers>=5.20.2" accelerate datasets librosa soundfile

Load the model and processor with trust_remote_code=True. The processor bundles the audio codec, which turns generated speech codes into a 24 kHz waveform. Access it as processor.audio_tokenizer and move it to the model’s device. Keep the codec in its own precision; don’t cast it to bfloat16.

import torch
from transformers import AutoModelForMultimodalLM, AutoProcessor

model_id = "LiquidAI/LFM3-Audio-2B"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32

model = AutoModelForMultimodalLM.from_pretrained(
    model_id, dtype=dtype, device_map=device, trust_remote_code=True
).eval()
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
codec = processor.audio_tokenizer.to(device=device).eval()

Output modes

Every task is a chat conversation, and the system prompt describes the task. Text output is the default: generation starts in text mode and switches to speech or interleaved output when the model emits an output-mode token. To select a mode from the start, prefill the assistant turn: put one of the prefixes below in a final assistant message and let the model continue it.

Assistant prefill	Behavior
None	Start in text mode. The model can still emit a token that switches modes.
`<\|audio_start\|>`	Start speech-only output, for example text-to-speech, until an end-of-audio frame.
`<\|mixed_start\|>`	Start interleaved text and speech, alternating blocks of 6 text tokens and 12 speech frames.
`<think>`	Begin a thinking block. The answer follows the closing `</think>`.
`<think></think>`	Supply an empty thinking block, so the answer starts immediately.

The thinking prefixes are for text-only answers. For those, also pass suppress_mixed=True and suppress_audio_out=True to generate(), so the model doesn’t switch to speech.

Helper

The helper below runs one assistant turn and returns the text and the speech waveform, if any. When you pass prefill, it sets continue_final_message=True, so generation continues the prefilled message instead of starting a new assistant turn. For speech and interleaved output, pass SPEECH_SAMPLING. These values are also the defaults in the model’s generation_config.json; passing them explicitly keeps the examples reproducible.

import soundfile as sf

SAMPLE_RATE_OUT = 24_000  # the codec decodes to 24 kHz
SPEECH_SAMPLING = dict(audio_temperature=1.0, audio_top_k=128, audio_repetition_penalty=1.1, audio_repetition_window=4096)


@torch.inference_mode()
def generate(conversation, prefill=None, max_new_tokens=512, **generate_kwargs):
    """Runs one assistant turn. Returns (text, waveform or None)."""
    if prefill is not None:
        conversation = conversation + [{"role": "assistant", "content": prefill}]
    inputs = processor.apply_chat_template(
        conversation,
        add_generation_prompt=prefill is None,
        continue_final_message=prefill is not None,
        tokenize=True,
        return_dict=True,
        return_tensors="pt",
    ).to(device)
    outputs = model.generate(**inputs, max_new_tokens=max_new_tokens, **generate_kwargs)
    text = processor.decode(outputs.sequences[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True)
    waveform = None
    if outputs.audio_codes is not None and outputs.audio_lengths[0] > 0:
        codes = outputs.audio_codes[:1, :, : outputs.audio_lengths[0]]
        waveform = codec.decode(codes).audio_values[0].float().cpu().numpy().squeeze()
    return text, waveform

Speech recognition

Use the system prompt Perform ASR in <language>. with the English name of the language, or Perform ASR. to let the model detect it. Pass audio in the user turn as {"type": "audio", "audio": ...}, where the value is a file path or URL. The processor resamples it to 16 kHz mono. This example transcribes a LibriSpeech clip:

import io
from datasets import Audio as AudioFeature, load_dataset

librispeech = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
librispeech = librispeech.cast_column("audio", AudioFeature(decode=False))

sample = librispeech[0]
speech, sr = sf.read(io.BytesIO(sample["audio"]["bytes"]))
sf.write("librispeech.wav", speech, sr)
print("Reference:", sample["text"])

transcript, _ = generate([
    {"role": "system", "content": "Perform ASR in English."},
    {"role": "user", "content": [{"type": "audio", "audio": "librispeech.wav"}]},
])
print("Transcript:", transcript)

Output

Reference: MISTER QUILTER IS THE APOSTLE OF THE MIDDLE CLASSES AND WE ARE GLAD TO WELCOME HIS GOSPEL
Transcript: Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.

Live speech recognition

Live speech recognition transcribes speech while it is still being spoken, instead of waiting for the full recording. Use the system prompt Perform live ASR.. It currently supports English only. The model processes audio in chunks of about 0.96 s and writes the text for each chunk as soon as the chunk arrives. model.transcribe_live() runs one chunk at a time and returns a state that you pass to the next call. In an application, you open a microphone stream into the model and a text stream out of it. This example simulates the microphone by feeding the LibriSpeech clip in 100 ms pieces:

import librosa

tokenizer = processor.tokenizer
chunk_frames = 8 * model.config.interleaved_n_audio
prompt = tokenizer(
    "<|startoftext|><|im_start|>system\nPerform live ASR.<|im_end|>\n<|im_start|>assistant\n",
    add_special_tokens=False,
    return_tensors="pt",
).input_ids.to(device)
suppress = tokenizer.convert_tokens_to_ids(["<|mixed_end|>", "<|endoftext|>"])
stop = tokenizer.convert_tokens_to_ids(["<|endoftext|>", "<|im_start|>"])

state, feature_cache = None, None
mel = torch.empty((1, processor.feature_extractor.feature_size, 0), device=device)


def transcribe(frames, is_final=False):
    global state
    output = model.transcribe_live(
        frames,
        state,
        input_ids=prompt if state is None else None,
        is_final=is_final,
        max_new_tokens=96 if is_final else 48,
        suppress_token_ids=suppress,
        stop_token_ids=stop,
    )
    state = output.state
    print(tokenizer.decode(output.sequences[0], skip_special_tokens=True), end="", flush=True)


waveform, _ = librosa.load("librispeech.wav", sr=16_000)
for start in range(0, len(waveform), 1_600):  # 100 ms microphone chunks
    features, feature_cache = processor.feature_extractor.stream(
        waveform[start:start + 1_600], feature_cache
    )
    mel = torch.cat((mel, features.input_features.to(device)), dim=-1)
    while mel.shape[-1] >= chunk_frames:
        transcribe(mel[..., :chunk_frames])
        mel = mel[..., chunk_frames:]

# Flush the remaining audio and end the transcript
features, feature_cache = processor.feature_extractor.stream([], feature_cache, is_final=True)
mel = torch.cat((mel, features.input_features.to(device)), dim=-1)
while mel.shape[-1] > chunk_frames:
    transcribe(mel[..., :chunk_frames])
    mel = mel[..., chunk_frames:]
transcribe(mel, is_final=True)
print()

Output

Mr. Quilter is the apostle of the middle classes, and we're glad to welcome his gospel.

For streams longer than 30 seconds, pass input_ids=prompt again to start a new text context. The encoder keeps following the audio stream.

Text-to-speech

Use the system prompt Perform TTS in <language> using a <female|male> voice. and prefill <|audio_start|>. Supported languages and accents: Arabic, Danish, German, Greek, British English, American English, Finnish, French, Hebrew, Italian, Japanese, Korean, Dutch, Polish, Portuguese, Spanish, Swedish, Turkish, and Chinese.

torch.manual_seed(0)
_, tts_en = generate(
    [
        {"role": "system", "content": "Perform TTS in American English using a female voice."},
        {"role": "user", "content": "I made a little plan for today: go for a walk, find a new workout playlist, and try a new cake recipe"},
    ],
    prefill="<|audio_start|>",
    **SPEECH_SAMPLING,
)
sf.write("tts_en.wav", tts_en, SAMPLE_RATE_OUT)

_, tts_fr = generate(
    [
        {"role": "system", "content": "Perform TTS in French using a male voice."},
        {"role": "user", "content": "On pourrait inventer une nouvelle recette ce soir. Si le résultat est étrange, on lui donnera un nom très chic."},
    ],
    prefill="<|audio_start|>",
    **SPEECH_SAMPLING,
)
sf.write("tts_fr.wav", tts_fr, SAMPLE_RATE_OUT)

As a quick check, let the model transcribe its own French speech:

transcript, _ = generate([
    {"role": "system", "content": "Perform ASR in French."},
    {"role": "user", "content": [{"type": "audio", "audio": "tts_fr.wav"}]},
])
print("Transcript:", transcript)

Output

Transcript: On pourrait inventer une nouvelle recette ce soir. Si le résultat est étrange, on lui donnera un nom très chic.

Speech translation

LFM3-Audio translates speech into text. Translation works best into or out of English. Name the source and target languages in the system prompt, for example Convert the speech in {source} to written {target}. or Translate the following speech from {source} to {target} in text form. The model was trained on many phrasings, so pick the one that works best for your data. For text-to-text translation, use the text model LFM3-1.7B instead. This example translates the French clip from the text-to-speech example into English:

translation, _ = generate([
    {"role": "system", "content": "Convert the speech in French to written English."},
    {"role": "user", "content": [{"type": "audio", "audio": "tts_fr.wav"}]},
])
print("Translation:", translation)

Output

Translation: We could invent a new recipe this evening. If the result is strange, it will be given a very fancy name.

Spoken chat

With the system prompt Respond with interleaved text and audio. and the <|mixed_start|> prefill, the model answers conversationally in text and speech at the same time, so speech can start playing before the full answer is written. Keep the system prompt exactly as shown, and give any further instructions in the user turn. This example first records a spoken question with text-to-speech, then asks it:

_, question = generate(
    [
        {"role": "system", "content": "Perform TTS in American English using a male voice."},
        {"role": "user", "content": "I want to move more, but I have not found an activity I enjoy. How can I try a few options and build a small habit?"},
    ],
    prefill="<|audio_start|>",
    **SPEECH_SAMPLING,
)
sf.write("question.wav", question, SAMPLE_RATE_OUT)

reply, reply_speech = generate(
    [
        {"role": "system", "content": "Respond with interleaved text and audio."},
        {"role": "user", "content": [{"type": "audio", "audio": "question.wav"}]},
    ],
    prefill="<|mixed_start|>",
    max_new_tokens=1024,
    **SPEECH_SAMPLING,
)
print("Text:", reply)
sf.write("reply.wav", reply_speech, SAMPLE_RATE_OUT)

Output (text part; the speech is saved to reply.wav)

Text: Try a short trial of each activity, like a weekend hike, a cooking class, or a volunteer shift, and note how much fun you had each time. Pick the one that feels most appealing and commit to doing it for just one week to see if it sticks. Does a one‑week trial of a single activity sound doable?

For streaming applications, model.generate() accepts streamer for text tokens and audio_streamer for speech frames, and the codec decodes those frames incrementally, so speech can be played as it is generated.

Speech-to-text chat

In this mode, the model behaves like a text LLM that also accepts speech. No system prompt is needed. Prefill <think> to let the model reason first; it closes the reasoning with </think>, and the answer follows. Pass suppress_mixed=True and suppress_audio_out=True to keep the answer in text.

text, _ = generate(
    [{"role": "user", "content": [{"type": "audio", "audio": "question.wav"}]}],
    prefill="<think>",
    max_new_tokens=2048,
    suppress_mixed=True,
    suppress_audio_out=True,
)
thinking, _, answer = text.partition("</think>")
print("Thinking:", thinking.strip())
print("Answer:", answer.strip())

Output (truncated)

Thinking: We need to answer the question: "I want to move more, but I have not found an activity I enjoy. How can I try a few options and build a small habit?"

1.  **Analyze the User's Request:**
    *   **Goal:** Move more.
    *   **Problem:** Hasn't found an activity they enjoy.
    *   **Solution Needed:** How to try a few options and build a small habit.
...
Answer: It is completely normal to struggle to find an activity you enjoy when you want to move more. The key to overcoming this is to break the process into tiny, manageable steps and focus on the *process* rather than the *result*.
...

To skip thinking, prefill an empty <think></think> block, so the answer starts immediately:

answer, _ = generate(
    [{"role": "user", "content": [{"type": "audio", "audio": "question.wav"}]}],
    prefill="<think></think>",
    max_new_tokens=1024,
    suppress_mixed=True,
    suppress_audio_out=True,
)
print("Answer:", answer.strip())

Audio understanding

Text and audio can be mixed in the same user turn, for example a recording plus a written question about it. Questions can address speech, music, background sounds, or events.

answer, _ = generate(
    [{"role": "user", "content": [
        {"type": "audio", "audio": "librispeech.wav"},
        {"type": "text", "text": "Who or what is the speaker talking about, and what is their attitude towards it?"},
    ]}],
    prefill="<think></think>",
    max_new_tokens=512,
    suppress_mixed=True,
    suppress_audio_out=True,
)
print("Answer:", answer.strip())

Output

Answer: The speaker is talking about **Mr. Quilter**, who is described as "the apostle of the middle classes." The speaker's attitude is one of welcome and approval—"we're glad to welcome his gospel," indicating a positive, supportive stance toward Mr. Quilter's message.

Tool calling

This example turns one spoken request into two function calls: cancel a streaming-platform subscription at the end of the billing period, then subscribe to a podcast platform’s Max plan with monthly billing. StreamBox and PodSpace are fictional services, and the example only prints the proposed calls. List the tools in the system prompt, between <tools> and </tools>, one JSON object per line. The user asks by voice, and the model returns one <tool_call> block per call.

import json

tools = [
    {
        "name": "cancel_subscription",
        "description": "Cancel the signed-in user's subscription to a streaming platform at the specified time.",
        "parameters": {
            "type": "object",
            "properties": {
                "platform": {"type": "string", "description": "Name of the streaming platform, for example StreamBox."},
                "effective_when": {
                    "type": "string",
                    "enum": ["immediately", "end_of_billing_period"],
                },
            },
            "required": ["platform", "effective_when"],
        },
    },
    {
        "name": "subscribe_to_podcast_platform",
        "description": "Subscribe the signed-in user to a podcast platform with the chosen plan and billing frequency.",
        "parameters": {
            "type": "object",
            "properties": {
                "platform": {"type": "string", "description": "Name of the podcast platform, for example PodSpace."},
                "plan": {"type": "string", "enum": ["basic", "plus", "max"]},
                "billing_frequency": {"type": "string", "enum": ["monthly", "annual"]},
            },
            "required": ["platform", "plan", "billing_frequency"],
        },
    },
]
tool_prompt = (
    "Return tool calls for both requested actions, in the order requested. "
    "Do not claim that an action has completed without a tool result.\n"
    "List of tools:\n<tools>\n" + "\n".join(json.dumps(t) for t in tools) + "\n</tools>"
)

# Record the spoken request
_, request = generate(
    [
        {"role": "system", "content": "Perform TTS in British English using a female voice."},
        {"role": "user", "content": "Please cancel my StreamBox subscription starting December first. Then subscribe me to PodSpace on the Max plan, billed monthly."},
    ],
    prefill="<|audio_start|>",
    **SPEECH_SAMPLING,
)
sf.write("request.wav", request, SAMPLE_RATE_OUT)

call, _ = generate(
    [
        {"role": "system", "content": tool_prompt},
        {"role": "user", "content": [{"type": "audio", "audio": "request.wav"}]},
    ],
    max_new_tokens=512,
    suppress_mixed=True,
    suppress_audio_out=True,
)
print(call)

Output

<tool_call>
<function=cancel_subscription>
<parameter=platform>
StreamBox
</parameter>
<parameter=effective_when>
end_of_billing_period
</parameter>
</function>
</tool_call>
<tool_call>
<function=subscribe_to_podcast_platform>
<parameter=platform>
PodSpace
</parameter>
<parameter=plan>
max
</parameter>
<parameter=billing_frequency>
monthly
</parameter>
</function>
</tool_call>

Generated calls use the LFM3 <tool_call> XML format. Your application parses and executes the calls and returns the results as tool messages. See Tool Use for the full workflow. Don’t prefill a thinking block for tool calls; the model calls tools without reasoning first. To get a spoken answer once the tool results are in, start the system prompt with Respond with function calls, and interleaved text and audio.\n\n, followed by any instructions and the tool list, and don’t pass suppress_mixed or suppress_audio_out. The model then either calls tools or answers in interleaved text and speech. The model card’s tool calling example shows the full loop, including returning tool results.

FAQ

Which model should I use? Use LFM3-Audio-2B for speech and audio tasks: transcription, speech synthesis, translation, spoken chat, audio understanding, and spoken tool calls. Use LFM3-Omni-2B when the request also involves images or video. See Omni Capabilities. What audio input is supported? Audio files are resampled to 16 kHz mono on load. Speech output is a 24 kHz waveform. Can the model stream speech? Yes. LFM3-Audio-2B uses a causal audio encoder that supports live audio input (see Live speech recognition), and its audio codec supports streaming decoding. In Transformers, pass streamer and audio_streamer to model.generate(). Does the model reason before answering? For text-only answers, prefill <think> to let the model reason first, or <think></think> to answer directly.

Models

Fine-tuning

Edge Inference

GPU Inference

Cloud Inference

Guides

Help

Setup

Output modes

Helper

Speech recognition

Live speech recognition

Text-to-speech

Speech translation

Spoken chat

Speech-to-text chat

Audio understanding

Tool calling

FAQ

​Setup

​Output modes

​Helper

​Speech recognition

​Live speech recognition

​Text-to-speech

​Speech translation

​Spoken chat

​Speech-to-text chat

​Audio understanding

​Tool calling

​FAQ

Setup

Output modes

Helper

Speech recognition

Live speech recognition

Text-to-speech

Speech translation

Spoken chat

Speech-to-text chat

Audio understanding

Tool calling

FAQ