llama.cpp - Liquid AI Docs

Use llama.cpp for CPU-only environments, local development, or edge deployment and on-device inference.

Building an application with llama.cpp? Jump to Build with llama.cpp for mobile, desktop, chat, tool-calling, structured-output, and multimodal guides.

For GPU-accelerated inference at scale, consider using vLLM instead.

Installation

Llama App
macOS/Linux
Pre-built Binaries
Build from Source

Install Llama, the desktop app from the llama.cpp team, for macOS and Windows. It runs llama.cpp with one-click model downloads, a built-in chat, and a local OpenAI-compatible API, with no command-line setup.

Install via Homebrew:

brew install llama.cpp

Download from llama.cpp releases.File naming: llama-<version>-bin-<os>-<feature>-<arch>.zipQuick selection guide:

Windows (CPU): llama-*-bin-win-avx2-x64.zip for Intel/AMD CPUs
Windows (NVIDIA GPU): llama-*-bin-win-cu12-x64.zip (requires CUDA drivers)
macOS (Intel): llama-*-bin-macos-x64.zip
macOS (Apple Silicon): llama-*-bin-macos-arm64.zip
Linux: llama-*-bin-linux-x64.zip

After downloading, unzip and run from that directory.

Detailed Download Tables by Platform

Use the tables below to determine which llama.cpp binary is best for your environment and download the relevant binary (version b7075), or browse all releases and find the latest version here.Windows

Hardware	Binary Name	Download Link
Nvidia GPU	llama-b7075-bin-win-cuda-12.4-x64.zip	Download
Intel GPU	llama-b7075-bin-win-sycl-x64.zip	Download
AMD GPU	llama-b7075-bin-win-vulkan-x64.zip	Download
Other GPU	llama-b7075-bin-win-vulkan-x64.zip	Download
Qualcomm Snapdragon CPU	llama-b7075-bin-win-cpu-arm64.zip	Download
Other (CPU-only)	llama-b7075-bin-win-cpu-x64.zip	Download

macOS

Hardware	Binary Name	Download Link
Intel	llama-b7075-bin-macos-x64.zip	Download
Apple Silicon	llama-b7075-bin-macos-arm64.zip	Download

Ubuntu

Hardware	Binary Name	Download Link
GPU	llama-b7075-bin-ubuntu-vulkan-x64.zip	Download
CPU-only	llama-b7075-bin-ubuntu-x64.zip	Download

Performance BenchmarksIf you are considering investing in hardware, here are some profiling results from a variety of machines and inference backends. As it currently stands, AMD Ryzen™ machines generally have the best-in-class performance with relatively standard llama.cpp configuration settings – and with custom configurations, this advantage tends to increase.

Device	Prefill speed (tok/s)	Decode speed (tok/s)
AMD Ryzen™ AI Max+ 395	5476	143
AMD Ryzen™ AI 9 HX 370	2680	113
Apple Mac Mini (M4)	1427	122
Qualcomm Snapdragon™ X1E-78-100	978	125
Intel Core™ Ultra 9 185H	1310	58
Intel Core™ Ultra 7 258V	1104	78

Note: for fair comparison, we conducted these benchmarks on the same model (LFM2-1.2B-Q4_0.gguf). For each hardware device, we also tested across all publicly available llama.cpp binaries, with different thread counts (4, 8, 12) for CPU runners, and took the best performing numbers for prefill and decode independently.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release -j 8

The compiled programs will be in ./build/bin/.For detailed build instructions including GPU support, see the llama.cpp documentation.

This guide uses llama.cpp v0.8.0 (b11960) or later, which LFM3 models require.

Downloading GGUF Models

llama.cpp uses the GGUF format, which stores quantized model weights for efficient inference. All LFM models are available in GGUF format on Hugging Face. See the Models page for all available GGUF models. You can download LFM models in GGUF format from Hugging Face as follows:

uv pip install huggingface-hub
hf download LiquidAI/LFM2.5-1.2B-Instruct-GGUF lfm2.5-1.2b-instruct-q4_k_m.gguf --local-dir .

Available quantization levels

Q4_0: 4-bit quantization, smallest size
Q4_K_M: 4-bit quantization, good balance of quality and size (recommended)
Q5_K_M: 5-bit quantization, better quality with moderate size increase
Q6_K: 6-bit quantization, excellent quality closer to original
Q8_0: 8-bit quantization, near-original quality
F16: 16-bit float, full precision

Basic Usage

llama.cpp offers two main interfaces for running inference: llama serve (web chat UI and OpenAI-compatible server) and llama cli (interactive CLI).

llama serve
llama cli

llama serve starts a web chat UI and an OpenAI-compatible API for serving models locally.Starting the Server:

llama serve -hf LiquidAI/LFM2.5-1.2B-Instruct-GGUF -c 4096

The -hf flag downloads the model directly from Hugging Face. Alternatively, use a local model file:

llama serve -m lfm2.5-1.2b-instruct-q4_k_m.gguf -c 4096

Key parameters:

-hf: Hugging Face model ID (downloads automatically)
-m: Path to local GGUF model file
-c: Context length (default: 4096)
--port: Server port (default: 9931)
-ngl 99: Offload layers to GPU (if available)

Using the Server:Open http://localhost:9931 in your browser to chat, or use the OpenAI Python client:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:9931/v1",
    api_key="not-needed"
)

response = client.chat.completions.create(
    model="lfm2.5-1.2b-instruct",
    messages=[
        {"role": "user", "content": "What is machine learning?"}
    ],
    temperature=0.1,
    max_tokens=512,
    extra_body={"top_k": 50, "repeat_penalty": 1.05},
)
print(response.choices[0].message.content)

Using curl:

curl http://localhost:9931/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "lfm2.5-1.2b-instruct",
    "messages": [{"role": "user", "content": "Hello!"}],
    "temperature": 0.1,
    "top_k": 50,
    "repeat_penalty": 1.05
  }'

llama cli provides an interactive terminal interface for chatting with models.

llama cli -hf LiquidAI/LFM2.5-1.2B-Instruct-GGUF -c 4096 \
    --temp 0.1 --top-k 50 --repeat-penalty 1.05

The -hf flag downloads the model directly from Hugging Face. Alternatively, use a local model file:

llama cli -m lfm2.5-1.2b-instruct-q4_k_m.gguf -c 4096 \
    --temp 0.1 --top-k 50 --repeat-penalty 1.05

Key parameters:

-hf: Hugging Face model ID (downloads automatically)
-m: Path to local GGUF model file
-c: Context length
-p: Prompt to start with. Add -st (--single-turn) to answer once and exit.
-ngl 99: Offload layers to GPU (if available)

Press Ctrl+C to exit.

Generation Parameters

Control text generation behavior using parameters in the OpenAI-compatible API or command-line flags. Key parameters:

temperature (float): Controls randomness (0.0 = deterministic, higher = more random). Typical range: 0.1-2.0
top_p (float): Nucleus sampling - limits to tokens with cumulative probability ≤ top_p. Typical range: 0.1-1.0
top_k (int): Limits to top-k most probable tokens. Typical range: 1-100
min_p (float): Filters tokens below min_p * max_probability. Typical range: 0.05-0.3
max_tokens / --n-predict (int): Maximum number of tokens to generate
repeat_penalty / --repeat-penalty (float): Penalty for repeating tokens (>1.0 = discourage repetition). Typical range: 1.0-1.5. The server ignores the repetition_penalty name used by vLLM and Transformers.
stop (str or list[str]): Strings that terminate generation when encountered

llama serve (OpenAI-compatible API) example

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:9931/v1",
    api_key="not-needed"
)

response = client.chat.completions.create(
    model="lfm2.5-1.2b-instruct",
    messages=[{"role": "user", "content": "What is machine learning?"}],
    temperature=0.1,
    max_tokens=512,
    extra_body={"top_k": 50, "repeat_penalty": 1.05},
)
print(response.choices[0].message.content)

For llama cli, use flags like --temp, --top-p, --top-k, --min-p, --repeat-penalty, and --n-predict.

Vision Models

LFM vision models (LFM3-VL and LFM2.5-VL) take images, and LFM3-VL also takes video. With -hf, llama.cpp downloads the matching vision projector (mmproj) automatically. The examples below use LFM3-VL-2B; the :Q4_0 tag selects its Quantization-Aware Distillation (QAD) checkpoint.

Command Line

llama cli -hf LiquidAI/LFM3-VL-2B-GGUF:Q4_0 \
  --image photo.jpg -p "Describe this image." \
  --top-p 1.0 --min-p 0.0 --no-reasoning-preserve

For video, use --video clip.mp4 instead of --image. Video input needs ffmpeg in PATH. Use --image-max-tokens to cap the number of tokens per image (lower is faster, coarser).

Server

llama serve -hf LiquidAI/LFM3-VL-2B-GGUF:Q4_0 \
  --top-p 1.0 --min-p 0.0 --no-reasoning-preserve

Send images as OpenAI image_url content parts, either a data: URI with base64 bytes or a public URL:

import base64
from openai import OpenAI

client = OpenAI(base_url="http://localhost:9931/v1", api_key="not-needed")

with open("photo.jpg", "rb") as f:
    image_b64 = base64.b64encode(f.read()).decode()

response = client.chat.completions.create(
    model="LiquidAI/LFM3-VL-2B-GGUF:Q4_0",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_b64}"}},
                {"type": "text", "text": "What's in this image?"},
            ],
        }
    ],
)
print(response.choices[0].message.content)

The LFM3 GGUFs store their recommended sampling parameters, so the request needs none. For LFM2.5-VL, see the sampling parameters on its model page.

Using local model files

Download the model and the projector, then pass both:

hf download LiquidAI/LFM3-VL-2B-GGUF LFM3-VL-2B-QAD-Q4_0.gguf mmproj-LFM3-VL-2B-Q8_0.gguf --local-dir .

llama serve -m LFM3-VL-2B-QAD-Q4_0.gguf --mmproj mmproj-LFM3-VL-2B-Q8_0.gguf \
  --top-p 1.0 --min-p 0.0 --no-reasoning-preserve

Audio and Omni Models

LFM3-Audio-2B and LFM3-Omni-2B take audio input, and LFM3-Omni-2B also takes images and video. With -hf, llama.cpp downloads the audio and vision projector automatically.

Speech Recognition

Pass Perform ASR in <language>. as a system prompt with -sys, and mark where the audio goes with -p "<__media__>". As a regular prompt, the model describes the audio instead of transcribing it.

llama cli -hf LiquidAI/LFM3-Audio-2B-GGUF:Q4_K_M \
  --audio speech.wav -sys "Perform ASR in English." -p "<__media__>"

To ask a question about a recording instead, drop the system prompt and put your question in -p.

Server

llama serve -hf LiquidAI/LFM3-Audio-2B-GGUF:Q4_K_M

For LFM3-Omni-2B, add --reasoning off. Send audio as OpenAI input_audio content parts (WAV, MP3, or FLAC):

import base64
from openai import OpenAI

client = OpenAI(base_url="http://localhost:9931/v1", api_key="not-needed")

with open("speech.wav", "rb") as f:
    audio_b64 = base64.b64encode(f.read()).decode()

response = client.chat.completions.create(
    model="LiquidAI/LFM3-Audio-2B-GGUF:Q4_K_M",
    messages=[
        {"role": "system", "content": "Perform ASR in English."},
        {"role": "user", "content": [{"type": "input_audio", "input_audio": {"data": audio_b64, "format": "wav"}}]},
    ],
)
print(response.choices[0].message.content)

To try it with your microphone, record a clip first, for example with arecord -f S16_LE -r 16000 -c 1 -d 5 speech.wav on Linux. For live audio, the server also handles real-time voice conversations, including tool calls, on its /v1/realtime WebSocket API.

Speech Synthesis

The llama command has no TTS subcommand, so speech synthesis uses the separate llama-tts program, which ships with llama.cpp:

hf download LiquidAI/LFM3-Audio-2B-GGUF LFM3-Audio-2B-Q4_K_M.gguf mmproj-LFM3-Audio-2B-Q8_0.gguf --local-dir .

llama-tts -m LFM3-Audio-2B-Q4_K_M.gguf -mm mmproj-LFM3-Audio-2B-Q8_0.gguf \
  -p "Hello world" --tts-voice male -o out.wav

llama serve also exposes a POST /tts endpoint. See the LFM3-Audio-2B and LFM3-Omni-2B pages for more examples, including image and video input with LFM3-Omni-2B.

Speculative Decoding

Some LFM2.5 models have DSpark draft models that make generation faster without changing the output. In llama.cpp, the drafter is a GGUF sidecar that you load next to the target with -hfd (or -md for a local file):

llama serve -hf LiquidAI/LFM2.5-1.2B-Instruct-GGUF:F16 \
  -hfd LiquidAI/LFM2.5-1.2B-Instruct-DSpark-GGUF:F16 \
  --spec-type draft-dspark --spec-draft-n-max 10 --spec-draft-n-min 0 \
  -ngl 99 -ngld 99 -fa on

The sidecar cannot run on its own. Speedups are largest with a 16-bit target. Each response’s timings report draft_n and draft_n_accepted. See Speculative Decoding for which models have drafters.

Converting Custom Models

If you have a finetuned model or need to create a GGUF from a Hugging Face model:

# Clone llama.cpp if you haven't already
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp

# Convert model with quantization
python convert_hf_to_gguf.py /path/to/your/model --outfile model.gguf --outtype q4_k_m

Use --outtype to specify the quantization level (e.g., q4_0, q4_k_m, q5_k_m, q6_k, q8_0, f16).

Building Applications

The guides below cover embedding llama.cpp in your own software — from a mobile app calling the C API to a desktop app driving the llama.cpp server:

iOS & Android

Link the XCFramework or NDK build and run LFM GGUFs in-process.

Desktop & Server Apps

llama-server as a sidecar, or Python / Node.js / .NET bindings.

Chat & Streaming

Multi-turn conversations, sampling parameters, prompt caching.

Function Calling & Agents

OpenAI-style tools with native LFM2.5 tool-call parsing.

Structured Output

JSON schema and GBNF grammar constrained generation.

Vision & Audio

LFM2.5-VL and LFM2.5-Audio in your own apps.

Example Applications

For more comprehensive example applications using llama.cpp with LFM models, check out these repositories:

The full list of llama.cpp language bindings can be found here.

​Installation

​Downloading GGUF Models

​Basic Usage

​Generation Parameters

​Vision Models

​Command Line

​Server

​Audio and Omni Models

​Speech Recognition

​Server

​Speech Synthesis

​Speculative Decoding

​Converting Custom Models

​Building Applications

iOS & Android

Desktop & Server Apps

Chat & Streaming

Function Calling & Agents

Structured Output

Vision & Audio

​Example Applications

Installation

Downloading GGUF Models

Basic Usage

Generation Parameters

Vision Models

Command Line

Server

Audio and Omni Models

Speech Recognition

Server

Speech Synthesis

Speculative Decoding

Converting Custom Models

Building Applications

Example Applications