# Baseten Source: https://docs.liquid.ai/deployment/gpu-inference/baseten Baseten is an AI infrastructure platform for deploying and serving ML models with optimized inference, autoscaling, and multi-cloud support. Use Baseten for production ML model deployments with optimized inference, autoscaling, and multi-cloud support. ## Clone the repository[​](#clone-the-repository "Direct link to Clone the repository") ``` git clone https://github.com/Liquid4All/lfm-inference ``` ## Deployment[​](#deployment "Direct link to Deployment") The deployment script is based on Baseten's [documentation](https://docs.baseten.co/examples/vllm) of `Run any LLM with vLLM`. Launch command: ``` cd bastenpip install trusstruss push lfm2-8b --publish ``` ## Test call[​](#test-call "Direct link to Test call") ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} curl -X POST https://.api.baseten.co/environments/production/predict \ -H "Authorization: Api-Key $BASETEN_API_KEY" \ -d '{ "model": "LiquidAI/LFM2-8B-A1B", "messages": [ { "role": "user", "content": "What is the melting temperature of silver?" } ], "max_tokens": 32, "temperature": 0 }' ``` Baseten endpoints expect the `Api-Key` prefix in the `Authorization` header. # Fal Source: https://docs.liquid.ai/deployment/gpu-inference/fal Fal is a serverless generative media platform offering lightning-fast inference for AI models for image, video, and audio generation. Use Fal for serverless cloud deployments with lightning-fast inference, autoscaling, and easy API access. ## Clone the repository ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} git clone https://github.com/Liquid4All/lfm-inference ``` ## Deployment ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} cd fal # run one-off server fal run deploy-lfm2.py::serve # run production server fal deploy deploy-lfm2.py::serve --app-name lfm2-8b --auth private ``` The first run will require extra time to download the docker image and model weights. ## Test call First, create an API key [here](https://fal.ai/dashboard/keys). Then run the following cURL commands: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} export FAL_API_KEY= # List deployed model curl https://fal.run///v1/models -H "Authorization: Key $FAL_API_KEY" # Query the deployed LFM model curl -X POST https://fal.run///v1/chat/completions \ -H "Authorization: Key $FAL_API_KEY" \ -d '{ "model": "LiquidAI/LFM2-8B-A1B", "messages": [ { "role": "user", "content": "What is the melting temperature of silver?" } ], "max_tokens": 32, "temperature": 0 }' ``` Fal endpoints expect the `Key` prefix in the `Authorization` header. # Modal Source: https://docs.liquid.ai/deployment/gpu-inference/modal Modal is a serverless cloud platform for running AI/ML workloads with instant autoscaling on GPUs and CPUs. Use Modal for serverless cloud deployments with instant autoscaling, GPU access, and production-ready inference serving. ## Clone the repository ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} git clone https://github.com/Liquid4All/lfm-inference ``` ## Deployment Launch command: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} cd modal # deploy LFM2 8B MoE model modal deploy deploy-vllm.py # deploy other LFM2 model, MODEL_NAME defaults to LiquidAI/LFM2-8B-A1B MODEL_NAME=LiquidAI/ modal deploy deploy-vllm.py ``` See full list of open source LFM models on [Hugging Face](https://huggingface.co/collections/LiquidAI/lfm2). ## Production deployment * Since vLLM takes over 2 min to cold start, if you run the inference server for production, it is recommended to keep a minimum number of warm instances with `min_containers = 1` and `buffer_containers = 1`. The `buffer_containers` config is necessary because all Modal GPUs are subject to [preemption](https://modal.com/docs/guide/preemption). See [docs](https://modal.com/docs/guide/cold-start#overprovision-resources-with-min_containers-and-buffer_containers) for details about cold start performance tuning. * Warm up the vLLM server after deployment by sending a single request. The warm-up process is included in the [deploy-vllm.py](https://github.com/Liquid4All/lfm-inference/blob/main/modal/deploy-vllm.py) script already. ## Test commands Test the deployed server with the following `curl` commands (replace `` with your actual deployment URL): ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} # List deployed model curl https:///v1/models # Query the deployed LFM model curl -X POST https:///v1/chat/completions \ -d '{ "model": "LiquidAI/LFM2-8B-A1B", "messages": [ { "role": "user", "content": "What is the melting temperature of silver?" } ], "max_tokens": 32, "temperature": 0 }' ``` # SGLang Source: https://docs.liquid.ai/deployment/gpu-inference/sglang Serve LFM2.5 with SGLang, a low-latency OpenAI-compatible serving framework. The full recipe, verified on real hardware, lives in the SGLang cookbook. SGLang serves LFM2.5 at low latency under high concurrency and exposes an OpenAI-compatible API. The dense, MoE, and vision-language models are supported natively, along with the `lfm2` tool-call parser and `` reasoning, so a single `sglang serve` puts any LFM2.5 checkpoint behind a production endpoint. Reach for SGLang when you want the lowest latency at high concurrency. It requires a CUDA GPU. On CPU-only machines, use [llama.cpp](/deployment/on-device/llama-cpp) instead. The SGLang cookbook generates the launch command for each model and GPU, including parsers and Blackwell attention backends. This page gets a server running. An interactive command generator for LFM2.5 on SGLang, verified on real hardware and updated with each release. ## What's in the cookbook * **A command for every model and GPU.** An interactive generator for the verified `sglang serve` line for your hardware. * **Reasoning and tool calling.** The right `--reasoning-parser` per model and `--tool-call-parser lfm2`, included in each command. * **Blackwell attention backends.** The `--attention-backend` and `--mm-attention-backend` choices that matter on B200/B300. * **Vision tuning.** Backends and memory flags for the VL models, with throughput numbers. * **Recommended sampling.** Per-checkpoint presets from the model cards. * **Benchmarks.** Measured TTFT, TPOT, and throughput per model and GPU. ## Quickstart Install SGLang from the [official guide](https://docs.sglang.io/get_started/install_sglang.html): ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} pip install --upgrade pip uv pip install sglang ``` Recent SGLang releases and the `lmsysorg/sglang` dev image include LFM2.5 support: the dense / MoE / VL model classes and the `lfm2` tool-call parser. On an older release, install from source or use the dev image. The cookbook lists the minimum version. Serve an LFM2.5 model behind an OpenAI-compatible API: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} sglang serve --model-path LiquidAI/LFM2.5-1.2B-Instruct --tool-call-parser lfm2 ``` This covers the dense Instruct model. The right `--reasoning-parser` for the thinking models, the vision and Blackwell attention backends, and the verified command for every model and GPU are generated by the [cookbook](https://docs.sglang.ai/cookbook/autoregressive/LiquidAI/LFM2.5). Use it for flags that change across releases. Send your first request with the OpenAI client: ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") response = client.chat.completions.create( model="LiquidAI/LFM2.5-1.2B-Instruct", messages=[{"role": "user", "content": "What is C. elegans? Answer in one sentence."}], temperature=0.1, extra_body={"top_k": 50, "repetition_penalty": 1.05}, ) print(response.choices[0].message.content) ``` ## Sampling parameters LFM2.5 uses per-model sampling presets. The [cookbook's configuration tips](https://docs.sglang.ai/cookbook/autoregressive/LiquidAI/LFM2.5#2-configuration-tips) list the exact values per checkpoint. Pass them on every request, since some checkpoints ship no defaults in `generation_config.json` and the server won't apply them for you. Put `top_k`, `min_p`, and `repetition_penalty` in `extra_body`, and leave `max_tokens` unset unless you mean to cap output. # Transformers Source: https://docs.liquid.ai/deployment/gpu-inference/transformers Transformers is a library for inference and training of pretrained models. Use Transformers for simple inference without extra dependencies, research and experimentation, or integration with the Hugging Face ecosystem. Transformers provides the most flexibility for model development and is ideal for users who want direct access to model internals. For production deployments with high throughput, consider using [vLLM](/deployment/gpu-inference/vllm). ## Installation We use [uv](https://docs.astral.sh/uv/#installation) to manage packages across all our code examples. It's backwards compatible with pip. Install the required dependencies: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} uv pip install "transformers>=5.2.0" torch accelerate ``` GPU is recommended for faster inference. ## Basic Usage The Transformers library provides two interfaces for text generation: [`generate()`](https://huggingface.co/docs/transformers/v4.57.1/en/main_classes/text_generation#transformers.GenerationMixin.generate) for fine-grained control and [`pipeline()`](https://huggingface.co/docs/transformers/en/main_classes/pipelines) for simplicity. We use `generate()` here for direct access to model internals and explicit control over the generation process: ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} from transformers import AutoModelForCausalLM, AutoTokenizer # Load model and tokenizer model_id = "LiquidAI/LFM2.5-1.2B-Instruct" model = AutoModelForCausalLM.from_pretrained( model_id, device_map="auto", dtype="bfloat16", # attn_implementation="flash_attention_2" <- uncomment on compatible GPU ) tokenizer = AutoTokenizer.from_pretrained(model_id) # Generate answer prompt = "What is C. elegans?" inputs = tokenizer.apply_chat_template( [{"role": "user", "content": prompt}], add_generation_prompt=True, return_tensors="pt", tokenize=True, return_dict=True, ).to(model.device) output = model.generate(**inputs, do_sample=True, temperature=0.1, top_k=50, repetition_penalty=1.05, max_new_tokens=512) # Decode only the newly generated tokens (excluding the input prompt) input_length = inputs["input_ids"].shape[1] response = tokenizer.decode(output[0][input_length:], skip_special_tokens=True) print(response) # C. elegans, also known as Caenorhabditis elegans, is a small, free-living # nematode worm (roundworm) that belongs to the phylum Nematoda. ``` **Model loading notes:** * **`model_id`**: Can be a Hugging Face model ID (e.g., `"LiquidAI/LFM2.5-1.2B-Instruct"`) or a local path * **`device_map="auto"`**: Automatically distributes across available GPUs/CPU (requires `accelerate`). Use `device="cuda"` for single GPU or `device="cpu"` for CPU only * **`dtype="bfloat16"`**: Recommended for modern GPUs. Use `"auto"` for automatic selection, or `"float32"` (slower, more memory) The [`pipeline()`](https://huggingface.co/docs/transformers/en/main_classes/pipelines) interface provides a simpler API for text generation with automatic chat template handling. It wraps model loading and tokenization, making it ideal for quick prototyping. ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} from transformers import pipeline generator = pipeline( "text-generation", "LiquidAI/LFM2.5-1.2B-Instruct", dtype="auto", device_map="auto", ) messages = [ {"role": "user", "content": "Give me a short introduction to large language models."}, ] messages = generator(messages, max_new_tokens=512)[0]["generated_text"] messages.append({"role": "user", "content": "In a single sentence."}) messages = generator(messages, max_new_tokens=512)[0]["generated_text"] ``` **Key parameters:** * **`"text-generation"`**: Task type for the pipeline * **`model_name_or_path`**: Model ID (e.g., `"LiquidAI/LFM2.5-1.2B-Instruct"`) or local path (download locally with `hf download --local-dir ./LFM2.5-1.2B-Instruct LiquidAI/LFM2.5-1.2B-Instruct`) * **`dtype="auto"`**: Automatically selects optimal dtype (`bfloat16` on modern devices). Can use `"bfloat16"` explicitly or `"float32"` (slower, more memory) * **`device_map="auto"`**: Automatically distributes across available GPUs/CPU (requires `accelerate`). Alternative: `device="cuda"` for single GPU, `device="cpu"` for CPU only. Don't mix `device_map` and `device` The pipeline automatically handles chat templates and tokenization, returning structured output with the generated text. ### Generation Parameters Control text generation behavior using [`GenerationConfig`](https://huggingface.co/docs/transformers/v4.57.1/en/main_classes/text_generation#transformers.GenerationConfig). Key parameters: * **`do_sample`** (`bool`): Enable sampling (`True`) or greedy decoding (`False`, default) * **`temperature`** (`float`, default 1.0): Controls randomness (0.0 = deterministic, higher = more random). Typical range: 0.1-2.0 * **`top_p`** (`float`, default 1.0): Nucleus sampling - limits to tokens with cumulative probability ≤ top\_p. Typical range: 0.1-1.0 * **`top_k`** (`int`, default 50): Limits to top-k most probable tokens. Typical range: 1-100 * **`min_p`** (`float`): Minimum token probability threshold. Typical range: 0.01-0.2 * **`max_new_tokens`** (`int`): Maximum number of tokens to generate (preferred over `max_length`) * **`repetition_penalty`** (`float`, default 1.0): Penalty for repeating tokens (>1.0 = discourage repetition). Typical range: 1.0-1.5 * **`stop_strings`** (`str` or `list[str]`): Strings that terminate generation when encountered Use `GenerationConfig` to organize parameters: ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} from transformers import GenerationConfig # Create a generation config generation_config = GenerationConfig( do_sample=True, temperature=0.1, top_k=50, repetition_penalty=1.05, max_new_tokens=512, ) # Use it in generate() output = model.generate(**inputs, generation_config=generation_config) ``` For a complete list of parameters, see the [GenerationConfig documentation](https://huggingface.co/docs/transformers/v4.57.1/en/main_classes/text_generation#transformers.GenerationConfig). ## Streaming Generation Stream responses as they're generated using `TextStreamer`: ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer # Use the model and tokenizer setup from Basic Usage above prompt = "Tell me a story about space exploration." inputs = tokenizer.apply_chat_template( [{"role": "user", "content": prompt}], add_generation_prompt=True, return_tensors="pt", tokenize=True, return_dict=True, ).to(model.device) streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True) output = model.generate(**inputs, do_sample=True, temperature=0.1, top_k=50, repetition_penalty=1.05, streamer=streamer, max_new_tokens=512) ``` ## Batch Generation Process multiple prompts in a single batch for efficiency. See the [batching documentation](https://huggingface.co/docs/transformers/en/main_classes/text_generation#batch-generation) for more details: Batching is not automatically a win for performance. For high-performance batching with optimized throughput, consider using [vLLM](/deployment/gpu-inference/vllm). ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} from transformers import AutoModelForCausalLM, AutoTokenizer # Use the model and tokenizer setup from Basic Usage above # Prepare multiple prompts prompts = [ [{"role": "user", "content": "Give me a short introduction to large language models."}], [{"role": "user", "content": "Give me a detailed introduction to large language models."}], ] # Apply chat templates and tokenize inputs = tokenizer.apply_chat_template( prompts, add_generation_prompt=True, return_tensors="pt", tokenize=True, padding=True, return_dict=True, ).to(model.device) # Generate for all prompts in batch outputs = model.generate(**inputs, do_sample=True, temperature=0.1, top_k=50, repetition_penalty=1.05, max_new_tokens=512) # Decode outputs for output in outputs: print(tokenizer.decode(output, skip_special_tokens=True)) ``` ## Vision Models LFM2-VL models support both text and images as input. Use `generate()` with the vision model and processor: ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} from transformers import AutoProcessor, AutoModelForImageTextToText from transformers.image_utils import load_image # Load model and processor model_id = "LiquidAI/LFM2.5-VL-1.6B" model = AutoModelForImageTextToText.from_pretrained( model_id, device_map="auto", dtype="bfloat16" ) processor = AutoProcessor.from_pretrained(model_id) # Load image and create conversation url = "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" image = load_image(url) # or use PIL Image: Image.open("path/to/image.jpg") conversation = [ { "role": "user", "content": [ {"type": "image", "image": image}, {"type": "text", "text": "What is in this image?"}, ], }, ] # Apply chat template and generate inputs = processor.apply_chat_template( conversation, add_generation_prompt=True, return_tensors="pt", return_dict=True, tokenize=True, ).to(model.device) outputs = model.generate(**inputs, do_sample=True, temperature=0.1, min_p=0.15, repetition_penalty=1.05, max_new_tokens=256) response = processor.batch_decode(outputs, skip_special_tokens=True)[0] print(response) ``` ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} from transformers import AutoProcessor, AutoModelForImageTextToText from transformers.image_utils import load_image # Use the model and processor setup from above image1 = load_image("path/to/first.jpg") image2 = load_image("path/to/second.jpg") conversation = [ { "role": "user", "content": [ {"type": "image", "image": image1}, {"type": "image", "image": image2}, {"type": "text", "text": "What are the differences between these two images?"} ], }, ] inputs = processor.apply_chat_template( conversation, add_generation_prompt=True, return_tensors="pt", return_dict=True, tokenize=True, ).to(model.device) outputs = model.generate(**inputs, do_sample=True, temperature=0.1, min_p=0.15, repetition_penalty=1.05, max_new_tokens=256) response = processor.batch_decode(outputs, skip_special_tokens=True)[0] print(response) ``` ## FAQ You may find distributed inference with Transformers is not as fast as you would imagine. Transformers with `device_map="auto"` does not apply tensor parallelism, and it only uses one GPU at a time. For Transformers with tensor parallelism, please refer to [its documentation](https://huggingface.co/docs/transformers/v4.51.3/en/perf_infer_gpu_multi). # vLLM Source: https://docs.liquid.ai/deployment/gpu-inference/vllm Serve LFM2.5 with vLLM, a high-throughput OpenAI-compatible inference engine. The full recipe, verified on real hardware, lives in the vLLM cookbook. vLLM is the highest-throughput way to serve LFM2.5 in production. Point vLLM at an LFM2.5 checkpoint to start an OpenAI-compatible endpoint with one command. The dense, MoE, and vision-language models are all native to vLLM (`Lfm2ForCausalLM`, `Lfm2MoeForCausalLM`, and `Lfm2VlForConditionalGeneration`), so there's no `--trust-remote-code` and nothing to patch. Reach for vLLM when you need high-throughput serving, batch processing, or an OpenAI-compatible API. It requires a CUDA GPU. On CPU-only machines, use [llama.cpp](/deployment/on-device/llama-cpp) instead. The vLLM cookbook keeps the serve flags, sampling presets, and tuning notes current. This page gets you from install to first request. Serving commands, sampling settings, and hardware notes for LFM2.5 on vLLM, verified on real hardware and updated with each release. ## What's in the cookbook * **Commands for every model.** The full LFM2.5 matrix (dense, MoE, reasoning, Japanese, base, and vision), with the exact command for each. * **Reasoning and tool calling.** When to pass `--reasoning-parser qwen3` and `--tool-call-parser lfm2`, and how the parsed output comes back. * **Recommended sampling.** Per-checkpoint presets from the model cards. * **Vision.** Serving the VL models and sending image-and-text turns. * **Blackwell tuning and benchmarks.** Throughput-impacting flags, with benchmark numbers. * **One-command Modal deploy.** A ready-to-run script for cloud GPUs. ## Quickstart Install vLLM. LFM2.5's dense, MoE, and VL architectures ship in **v0.23.0** and later: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} uv pip install -U vllm ``` Serve any LFM2.5 model behind an OpenAI-compatible API: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} vllm serve LiquidAI/LFM2.5-1.2B-Instruct ``` This is the minimal path. The per-model and per-hardware flags (`--tool-call-parser lfm2`, `--reasoning-parser qwen3`, vision options, and Blackwell tuning) live in the [cookbook](https://docs.vllm.ai/projects/recipes/en/latest/LiquidAI/LFM2.5.html), where they stay in sync with each release. Send your first request with the OpenAI client: ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") response = client.chat.completions.create( model="LiquidAI/LFM2.5-1.2B-Instruct", messages=[{"role": "user", "content": "What is C. elegans? Answer in one sentence."}], temperature=0.1, extra_body={"top_k": 50, "repetition_penalty": 1.05}, ) print(response.choices[0].message.content) ``` ## Sampling parameters LFM2.5 ships per-model sampling presets on its Hugging Face model cards. The [cookbook's sampling table](https://docs.vllm.ai/projects/recipes/en/latest/LiquidAI/LFM2.5.html#recommended-sampling) has the exact values for every checkpoint. Two details matter: `top_k`, `min_p`, and `repetition_penalty` are vLLM extras, so pass them through `extra_body` rather than as top-level arguments; and leave `max_tokens` unset unless you mean to cap output, since a low cap cuts off the reasoning models mid-thought. # Atomic Chat Source: https://docs.liquid.ai/deployment/on-device/atomic-chat Atomic Chat is a desktop and mobile app for running LLMs locally with a graphical user interface. Use Atomic Chat for local inference with a graphical interface on desktop and mobile, one-click model downloads from Hugging Face, and no command-line setup. Atomic Chat uses GGUF models on all platforms and MLX models on Apple Silicon. ## Installation Download and install Atomic Chat from [atomic.chat](https://atomic.chat): * **macOS** (Apple Silicon): DMG installer * **Windows** (x64): EXE installer * **Linux** (x86\_64): AppImage * **iPhone and iPad**: [App Store](https://apps.apple.com/us/app/atomic-chat-private-local-ai/id6761720226) * **Android**: [Google Play](https://play.google.com/store/apps/details?id=chat.atomic.app) ## Downloading Models 1. Open Atomic Chat and open the model library via the **Models** tab 2. Search for "LiquidAI" 3. Select a model and quantization level (`Q4_K_M` recommended) 4. Click **Download** Alternatively, enable [Hugging Face Local Apps](https://huggingface.co/docs/hub/local-apps), then choose **Use this model** > **Atomic Chat** from a compatible model page. See the [Models page](/lfm/models/complete-library) for all available GGUF models. ## Using the Chat Interface 1. Go to the **New Chat** tab 2. Select your model from the dropdown 3. Adjust parameters (`temperature`, `top_k`, `repeat_penalty`) in the model settings 4. Start chatting ## Generation Parameters Control text generation behavior using the GUI sidebar or API parameters. Key parameters: * **`temperature`** (`float`, default 1.0): Controls randomness (0.0 = deterministic, higher = more random). Typical range: 0.1-2.0 * **`top_p`** (`float`, default 1.0): Nucleus sampling - limits to tokens with cumulative probability ≤ top\_p. Typical range: 0.1-1.0 * **`top_k`** (`int`, default 40): Limits to top-k most probable tokens. Typical range: 1-100 * **`repeat_penalty`** (`float`, default 1.0): Penalty for repeating tokens (>1.0 = discourage repetition). Typical range: 1.0-1.5 Via the OpenAI-compatible API: ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} response = client.chat.completions.create( model="", messages=[{"role": "user", "content": "What is machine learning?"}], temperature=0.1, max_tokens=512, extra_body={"top_k": 50, "repeat_penalty": 1.05}, ) ``` ## Running the Server On desktop, Atomic Chat can serve the currently loaded model through a local OpenAI-compatible server for programmatic access: 1. Load the model in a chat 2. Open the **Integrations** tab 3. Click **Start Server**. The server defaults to `http://localhost:1337/`; use the port shown in the app if you changed it or if that port was unavailable. Get the loaded model's ID before sending requests: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} curl http://localhost:1337/v1/models ``` Use the OpenAI Python client: ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} from openai import OpenAI client = OpenAI( base_url="http://localhost:1337/v1", api_key="not-needed" ) response = client.chat.completions.create( model="", messages=[ {"role": "user", "content": "What is machine learning?"} ], temperature=0.1, max_tokens=512, extra_body={"top_k": 50, "repeat_penalty": 1.05}, ) print(response.choices[0].message.content) ``` ### Streaming Responses ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} stream = client.chat.completions.create( model="", messages=[ {"role": "user", "content": "Tell me a story."} ], stream=True ) for chunk in stream: if chunk.choices[0].delta.content is not None: print(chunk.choices[0].delta.content, end="") ``` You can also use curl to interact with the server: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} curl http://localhost:1337/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "", "messages": [{"role": "user", "content": "Hello!"}], "temperature": 0.1, "top_k": 50, "repeat_penalty": 1.05 }' ``` ## Vision Models Atomic Chat supports LFM2-VL and LFM2.5-VL GGUF models on desktop. Its mobile catalog is curated by platform; LFM2.5-VL-1.6B is available for mobile vision inference. Download a vision model from the model library, then attach images to your messages to ask questions about them. ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} from openai import OpenAI import base64 client = OpenAI( base_url="http://localhost:1337/v1", api_key="not-needed" ) # Encode image to base64 with open("image.jpg", "rb") as image_file: image_data = base64.b64encode(image_file.read()).decode("utf-8") response = client.chat.completions.create( model="", messages=[ { "role": "user", "content": [ {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_data}"}}, {"type": "text", "text": "What's in this image?"} ] } ] ) print(response.choices[0].message.content) ``` ## Tips * **Quantization**: `Q4_K_M` offers the best balance of size and quality; step up to `Q6_K` or `Q8_0` if you have memory to spare * **Apple Silicon**: GGUF models run with Metal acceleration, and MLX builds of LFM models are supported natively * **Long conversations (desktop)**: TurboQuant can compress the KV cache to 3 or 4 bits, so long contexts fit in significantly less memory # llama.cpp Source: https://docs.liquid.ai/deployment/on-device/llama-cpp llama.cpp is a C++ library for efficient LLM inference with minimal dependencies. It's designed for CPU-first inference with cross-platform support. Use llama.cpp for CPU-only environments, local development, or edge deployment and on-device inference. Building an application with llama.cpp? Jump to [Build with llama.cpp](#building-applications) for mobile, desktop, chat, tool-calling, structured-output, and multimodal guides. For GPU-accelerated inference at scale, consider using [vLLM](/deployment/gpu-inference/vllm) instead. ## Installation Install via Homebrew: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} brew install llama.cpp ``` Download from [llama.cpp releases](https://github.com/ggml-org/llama.cpp/releases). **File naming:** `llama--bin---.zip` **Quick selection guide:** * **Windows (CPU)**: `llama-*-bin-win-avx2-x64.zip` for Intel/AMD CPUs * **Windows (NVIDIA GPU)**: `llama-*-bin-win-cu12-x64.zip` (requires CUDA drivers) * **macOS (Intel)**: `llama-*-bin-macos-x64.zip` * **macOS (Apple Silicon)**: `llama-*-bin-macos-arm64.zip` * **Linux**: `llama-*-bin-linux-x64.zip` After downloading, unzip and run from that directory. Use the tables below to determine which `llama.cpp` binary is best for your environment and download the relevant binary (version `b7075`), or browse all releases and find the latest version [here](https://github.com/ggml-org/llama.cpp/releases). **Windows** | Hardware | Binary Name | Download Link | | ----------------------- | ------------------------------------- | --------------------------------------------------------------------------------------------------------------- | | Nvidia GPU | llama-b7075-bin-win-cuda-12.4-x64.zip | [Download](https://github.com/ggml-org/llama.cpp/releases/download/b7075/llama-b7075-bin-win-cuda-12.4-x64.zip) | | Intel GPU | llama-b7075-bin-win-sycl-x64.zip | [Download](https://github.com/ggml-org/llama.cpp/releases/download/b7075/llama-b7075-bin-win-sycl-x64.zip) | | AMD GPU | llama-b7075-bin-win-vulkan-x64.zip | [Download](https://github.com/ggml-org/llama.cpp/releases/download/b7075/llama-b7075-bin-win-vulkan-x64.zip) | | Other GPU | llama-b7075-bin-win-vulkan-x64.zip | [Download](https://github.com/ggml-org/llama.cpp/releases/download/b7075/llama-b7075-bin-win-vulkan-x64.zip) | | Qualcomm Snapdragon CPU | llama-b7075-bin-win-cpu-arm64.zip | [Download](https://github.com/ggml-org/llama.cpp/releases/download/b7075/llama-b7075-bin-win-cpu-arm64.zip) | | Other (CPU-only) | llama-b7075-bin-win-cpu-x64.zip | [Download](https://github.com/ggml-org/llama.cpp/releases/download/b7075/llama-b7075-bin-win-cpu-x64.zip) | **macOS** | Hardware | Binary Name | Download Link | | ------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------- | | Intel | llama-b7075-bin-macos-x64.zip | [Download](https://github.com/ggml-org/llama.cpp/releases/download/b7075/llama-b7075-bin-macos-x64.zip) | | Apple Silicon | llama-b7075-bin-macos-arm64.zip | [Download](https://github.com/ggml-org/llama.cpp/releases/download/b7075/llama-b7075-bin-macos-arm64.zip) | **Ubuntu** | Hardware | Binary Name | Download Link | | -------- | ------------------------------------- | --------------------------------------------------------------------------------------------------------------- | | GPU | llama-b7075-bin-ubuntu-vulkan-x64.zip | [Download](https://github.com/ggml-org/llama.cpp/releases/download/b7075/llama-b7075-bin-ubuntu-vulkan-x64.zip) | | CPU-only | llama-b7075-bin-ubuntu-x64.zip | [Download](https://github.com/ggml-org/llama.cpp/releases/download/b7075/llama-b7075-bin-ubuntu-x64.zip) | **Performance Benchmarks** If you are considering investing in hardware, here are some profiling results from a variety of machines and inference backends. As it currently stands, AMD Ryzen™ machines generally have the best-in-class performance with relatively standard llama.cpp configuration settings – and with custom configurations, this advantage tends to increase. | Device | Prefill speed (tok/s) | Decode speed (tok/s) | | ------------------------------- | --------------------- | -------------------- | | AMD Ryzen™ AI Max+ 395 | 5476 | 143 | | AMD Ryzen™ AI 9 HX 370 | 2680 | 113 | | Apple Mac Mini (M4) | 1427 | 122 | | Qualcomm Snapdragon™ X1E-78-100 | 978 | 125 | | Intel Core™ Ultra 9 185H | 1310 | 58 | | Intel Core™ Ultra 7 258V | 1104 | 78 | Note: for fair comparison, we conducted these benchmarks on the same model (`LFM2-1.2B-Q4_0.gguf`). For each hardware device, we also tested across all publicly available llama.cpp binaries, with different thread counts (4, 8, 12) for CPU runners, and took the best performing numbers for prefill and decode independently. ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} git clone https://github.com/ggml-org/llama.cpp cd llama.cpp cmake -B build cmake --build build --config Release -j 8 ``` The compiled programs will be in `./build/bin/`. For detailed build instructions including GPU support, see the [llama.cpp documentation](https://github.com/ggerganov/llama.cpp#build). ## Downloading GGUF Models llama.cpp uses the GGUF format, which stores quantized model weights for efficient inference. All LFM models are available in GGUF format on Hugging Face. See the [Models page](/lfm/models/complete-library) for all available GGUF models. You can download LFM models in GGUF format from Hugging Face as follows: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} uv pip install huggingface-hub hf download LiquidAI/LFM2.5-1.2B-Instruct-GGUF lfm2.5-1.2b-instruct-q4_k_m.gguf --local-dir . ``` * `Q4_0`: 4-bit quantization, smallest size * `Q4_K_M`: 4-bit quantization, good balance of quality and size (recommended) * `Q5_K_M`: 5-bit quantization, better quality with moderate size increase * `Q6_K`: 6-bit quantization, excellent quality closer to original * `Q8_0`: 8-bit quantization, near-original quality * `F16`: 16-bit float, full precision ## Basic Usage llama.cpp offers two main interfaces for running inference: `llama-server` (OpenAI-compatible server) and `llama-cli` (interactive CLI). llama-server provides an OpenAI-compatible API for serving models locally. **Starting the Server:** ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} llama-server -hf LiquidAI/LFM2.5-1.2B-Instruct-GGUF -c 4096 --port 8080 ``` The `-hf` flag downloads the model directly from Hugging Face. Alternatively, use a local model file: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} llama-server -m lfm2.5-1.2b-instruct-q4_k_m.gguf -c 4096 --port 8080 ``` Key parameters: * `-hf`: Hugging Face model ID (downloads automatically) * `-m`: Path to local GGUF model file * `-c`: Context length (default: 4096) * `--port`: Server port (default: 8080) * `-ngl 99`: Offload layers to GPU (if available) **Using the Server:** Once running at `http://localhost:8080`, use the OpenAI Python client: ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} from openai import OpenAI client = OpenAI( base_url="http://localhost:8080/v1", api_key="not-needed" ) response = client.chat.completions.create( model="lfm2.5-1.2b-instruct", messages=[ {"role": "user", "content": "What is machine learning?"} ], temperature=0.1, max_tokens=512, extra_body={"top_k": 50, "repetition_penalty": 1.05}, ) print(response.choices[0].message.content) ``` **Using curl:** ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} curl http://localhost:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "lfm2.5-1.2b-instruct", "messages": [{"role": "user", "content": "Hello!"}], "temperature": 0.1, "top_k": 50, "repetition_penalty": 1.05 }' ``` llama-cli provides an interactive terminal interface for chatting with models. ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} llama-cli -hf LiquidAI/LFM2.5-1.2B-Instruct-GGUF -c 4096 --color -i \ --temp 0.1 --top-k 50 --repeat-penalty 1.05 ``` The `-hf` flag downloads the model directly from Hugging Face. Alternatively, use a local model file: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} llama-cli -m lfm2.5-1.2b-instruct-q4_k_m.gguf -c 4096 --color -i \ --temp 0.1 --top-k 50 --repeat-penalty 1.05 ``` Key parameters: * `-hf`: Hugging Face model ID (downloads automatically) * `-m`: Path to local GGUF model file * `-c`: Context length * `--color`: Colored output * `-i`: Interactive mode * `-ngl 99`: Offload layers to GPU (if available) Press Ctrl+C to exit. ## Generation Parameters Control text generation behavior using parameters in the OpenAI-compatible API or command-line flags. Key parameters: * **`temperature`** (`float`): Controls randomness (0.0 = deterministic, higher = more random). Typical range: 0.1-2.0 * **`top_p`** (`float`): Nucleus sampling - limits to tokens with cumulative probability ≤ top\_p. Typical range: 0.1-1.0 * **`top_k`** (`int`): Limits to top-k most probable tokens. Typical range: 1-100 * **`min_p`** (`float`): Filters tokens below `min_p * max_probability`. Typical range: 0.05-0.3 * **`max_tokens`** / **`--n-predict`** (`int`): Maximum number of tokens to generate * **`repetition_penalty`** / **`--repeat-penalty`** (`float`): Penalty for repeating tokens (>1.0 = discourage repetition). Typical range: 1.0-1.5 * **`stop`** (`str` or `list[str]`): Strings that terminate generation when encountered ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} from openai import OpenAI client = OpenAI( base_url="http://localhost:8080/v1", api_key="not-needed" ) response = client.chat.completions.create( model="lfm2.5-1.2b-instruct", messages=[{"role": "user", "content": "What is machine learning?"}], temperature=0.1, max_tokens=512, extra_body={"top_k": 50, "repetition_penalty": 1.05}, ) print(response.choices[0].message.content) ``` For command-line tools (`llama-cli`), use flags like `--temp`, `--top-p`, `--top-k`, `--min-p`, `--repeat-penalty`, and `--n-predict`. ## Vision Models LFM2-VL GGUF models can be used for multimodal inference with llama.cpp. ### Quick Start with llama-cli Download llama.cpp binaries and run vision inference directly: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} wget https://github.com/ggml-org/llama.cpp/releases/download/b7633/llama-b7633-bin-ubuntu-x64.tar.gz tar -xzf llama-b7633-bin-ubuntu-x64.tar.gz ``` Download a test image: ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} import requests image_url = "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" img_data = requests.get(image_url).content with open("test_image.jpg", "wb") as f: f.write(img_data) ``` Run inference (works on CPU): ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} llama-b7633/llama-cli \ -hf LiquidAI/LFM2.5-VL-1.6B-GGUF:Q4_0 \ --image test_image.jpg \ --image-max-tokens 64 \ -p "What's in this image?" \ -n 128 \ --temp 0.1 --min-p 0.15 --repeat-penalty 1.05 ``` The `-hf` flag downloads the model directly from Hugging Face. Use `--image-max-tokens` to control image token budget. ### Alternative: Manual Model Download If you prefer to download models manually: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} uv pip install huggingface-hub hf download LiquidAI/LFM2-VL-1.6B-GGUF LFM2-VL-1.6B-Q8_0.gguf --local-dir . hf download LiquidAI/LFM2-VL-1.6B-GGUF mmproj-LFM2-VL-1.6B-Q8_0.gguf --local-dir . ``` Run inference directly from the command line: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} llama-mtmd-cli \ -m LFM2-VL-1.6B-Q8_0.gguf \ --mmproj mmproj-LFM2-VL-1.6B-Q8_0.gguf \ --image image.jpg \ -p "What is in this image?" \ -ngl 99 ``` Start a vision model server with both the model and mmproj files: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} llama-server \ -m LFM2-VL-1.6B-Q8_0.gguf \ --mmproj mmproj-LFM2-VL-1.6B-Q8_0.gguf \ -c 4096 \ --port 8080 \ -ngl 99 ``` Use with the OpenAI Python client: ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} from openai import OpenAI import base64 client = OpenAI( base_url="http://localhost:8080/v1", # The hosted llama-server api_key="not-needed" ) # Encode image to base64 with open("image.jpg", "rb") as image_file: image_data = base64.b64encode(image_file.read()).decode("utf-8") response = client.chat.completions.create( model="lfm2.5-vl-1.6b", # Model name should match your server configuration messages=[ { "role": "user", "content": [ {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_data}"}}, {"type": "text", "text": "What's in this image?"} ] } ], temperature=0.1, max_tokens=256, extra_body={"min_p": 0.15, "repetition_penalty": 1.05}, ) print(response.choices[0].message.content) ``` For a complete working example with step-by-step instructions, see the [llama.cpp Vision Model Colab notebook](https://colab.research.google.com/drive/1q2PjE6O_AahakRlkTNJGYL32MsdUcj7b?usp=sharing). ## Converting Custom Models If you have a finetuned model or need to create a GGUF from a Hugging Face model: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} # Clone llama.cpp if you haven't already git clone https://github.com/ggerganov/llama.cpp cd llama.cpp # Convert model with quantization python convert_hf_to_gguf.py /path/to/your/model --outfile model.gguf --outtype q4_k_m ``` Use `--outtype` to specify the quantization level (e.g., `q4_0`, `q4_k_m`, `q5_k_m`, `q6_k`, `q8_0`, `f16`). ## Building Applications The guides below cover embedding llama.cpp in your own software — from a mobile app calling the C API to a desktop app driving `llama-server`: Link the XCFramework or NDK build and run LFM GGUFs in-process. llama-server as a sidecar, or Python / Node.js / .NET bindings. Multi-turn conversations, sampling parameters, prompt caching. OpenAI-style tools with native LFM2.5 tool-call parsing. JSON schema and GBNF grammar constrained generation. LFM2.5-VL and LFM2.5-Audio on llama.cpp. ## Example Applications For more comprehensive example applications using llama.cpp with LFM models, check out these repositories: * [JavaScript (NodeJS) example](https://github.com/Liquid4All/leap-llamacpp-electron-example) * [Python example](https://github.com/Liquid4All/leap-llamacpp-python-example) * [C# example](https://github.com/Liquid4All/leap-llamacpp-csharp-example) The full list of llama.cpp language bindings can be found [here](https://github.com/ggml-org/llama.cpp?tab=readme-ov-file#description). # iOS & Android Source: https://docs.liquid.ai/deployment/on-device/llama-cpp/mobile Embed llama.cpp directly in an iOS or Android app and run LFM GGUF models on-device through the native C API. llama.cpp is a dependency-free C/C++ library, so it links straight into a mobile app. Every LFM checkpoint ships as GGUF on Hugging Face ([LiquidAI](https://huggingface.co/LiquidAI)), and upstream llama.cpp supports the LFM2 architecture, LFM2-VL projectors, and LFM2/LFM2.5 tool-call parsing. No wrapper SDK is required. On a phone, run the library **in-process** through the C API as shown here. `llama-server` is the right tool on laptops, desktops, and servers — see [Desktop & Server Apps](/deployment/on-device/llama-cpp/desktop). ## 1. Add llama.cpp to your project Every llama.cpp release publishes a prebuilt `llama.xcframework` with slices for iOS (device and simulator), macOS, visionOS, and tvOS. It is built with Metal enabled and includes the `mtmd` multimodal library. 1. Download `llama--xcframework.zip` from [llama.cpp releases](https://github.com/ggml-org/llama.cpp/releases) and unzip it. 2. In Xcode, drag `llama.xcframework` into your target's **Frameworks, Libraries, and Embedded Content**. 3. `import llama` in Swift. The C API is exposed directly; no bridging header is needed. The prebuilt framework targets iOS 16.4+ and macOS 13.3+. To build it yourself (for example, to change the minimum OS version or drop slices): ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} git clone https://github.com/ggml-org/llama.cpp cd llama.cpp ./build-xcframework.sh # output: build-apple/llama.xcframework ``` The upstream [`llama.swiftui`](https://github.com/ggml-org/llama.cpp/tree/master/examples/llama.swiftui) example is a complete SwiftUI chat app built this way. llama.cpp ships an Android Studio project at [`examples/llama.android`](https://github.com/ggml-org/llama.cpp/tree/master/examples/llama.android). Its `lib` module compiles llama.cpp with CMake through the NDK, includes CPU kernels up to Arm SME2 with runtime feature detection, and exposes a small Kotlin API (`AiChat` / `InferenceEngine`). Import the directory into Android Studio, run a Gradle sync, and depend on the `lib` module from your app (or copy it into your project). If you prefer to own the JNI layer, add llama.cpp as a CMake subdirectory of your native module: ```cmake theme={"theme":{"light":"github-light","dark":"github-dark"}} # app/src/main/cpp/CMakeLists.txt cmake_minimum_required(VERSION 3.22) project(myapp) set(LLAMA_BUILD_COMMON OFF) set(LLAMA_BUILD_TESTS OFF) set(LLAMA_BUILD_EXAMPLES OFF) set(LLAMA_BUILD_TOOLS OFF) set(LLAMA_BUILD_SERVER OFF) add_subdirectory(${CMAKE_CURRENT_SOURCE_DIR}/llama.cpp build-llama) add_library(myapp SHARED myapp.cpp) target_link_libraries(myapp llama android log) ``` ```kotlin theme={"theme":{"light":"github-light","dark":"github-dark"}} // app/build.gradle.kts android { defaultConfig { ndk { abiFilters += listOf("arm64-v8a") } } externalNativeBuild { cmake { path = file("src/main/cpp/CMakeLists.txt") } } } ``` Prebuilt `llama--bin-android-arm64.tar.gz` archives on the [releases page](https://github.com/ggml-org/llama.cpp/releases) contain `llama-cli`, `llama-server`, and `llama-bench` for quick testing on a device over `adb` or in Termux. See [docs/android.md](https://github.com/ggml-org/llama.cpp/blob/master/docs/android.md) for the Termux and NDK cross-compile recipes. ## 2. Get a model onto the device Download a GGUF from Hugging Face at first launch and keep it in app-private storage. llama.cpp memory-maps the file, so it must be a real file on disk — not a compressed asset. ``` https://huggingface.co/LiquidAI/LFM2.5-1.2B-Instruct-GGUF/resolve/main/LFM2.5-1.2B-Instruct-Q4_0.gguf ``` * Use `URLSessionConfiguration.background(withIdentifier:)` on iOS and `WorkManager` (or `DownloadManager`) on Android so downloads survive backgrounding. * **`Q4_0`** is the best default on phones: it is the smallest quantization and llama.cpp repacks it into Arm-optimized kernels at load time. Use `Q4_K_M` when you want slightly better quality on capable devices. See [Model Library](/lfm/models/complete-library) for every GGUF repository. * For vision models also download the matching `mmproj-*.gguf` from the same repository (see [Vision & Audio](/deployment/on-device/llama-cpp/multimodal)). For development you can push a file directly: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} uv pip install huggingface-hub hf download LiquidAI/LFM2.5-1.2B-Instruct-GGUF LFM2.5-1.2B-Instruct-Q4_0.gguf --local-dir . adb push LFM2.5-1.2B-Instruct-Q4_0.gguf /data/local/tmp/ # Android ``` ## 3. Load the model and stream a response The generation loop is the same on every platform: load the model, create a context, build a sampler chain with the model's [sampling parameters](/deployment/on-device/llama-cpp/chat#sampling-parameters), format the prompt with the [chat template](/lfm/key-concepts/chat-template), then decode and sample one token at a time. ```swift theme={"theme":{"light":"github-light","dark":"github-dark"}} import Foundation import llama enum RunnerError: Error { case modelLoadFailed, contextInitFailed } /// Minimal llama.cpp runner for LFM2.5-1.2B-Instruct. final class LFMRunner { private let model: OpaquePointer private let ctx: OpaquePointer private let vocab: OpaquePointer private let sampler: OpaquePointer init(modelPath: String, contextLength: UInt32 = 4096) throws { llama_backend_init() var modelParams = llama_model_default_params() modelParams.n_gpu_layers = 99 // offload to Metal; set 0 for CPU-only guard let model = llama_model_load_from_file(modelPath, modelParams) else { throw RunnerError.modelLoadFailed } self.model = model self.vocab = llama_model_get_vocab(model) var ctxParams = llama_context_default_params() ctxParams.n_ctx = contextLength ctxParams.n_batch = 512 ctxParams.n_threads = Int32(max(1, ProcessInfo.processInfo.activeProcessorCount - 2)) ctxParams.n_threads_batch = ctxParams.n_threads guard let ctx = llama_init_from_model(model, ctxParams) else { throw RunnerError.contextInitFailed } self.ctx = ctx // LFM2.5-1.2B-Instruct: temperature 0.1, top_k 50, repetition penalty 1.05 let sampler = llama_sampler_chain_init(llama_sampler_chain_default_params()) llama_sampler_chain_add(sampler, llama_sampler_init_top_k(50)) llama_sampler_chain_add(sampler, llama_sampler_init_penalties( llama_vocab_n_tokens(vocab), 64, 1.05, 0.0, 0.0)) llama_sampler_chain_add(sampler, llama_sampler_init_temp(0.1)) llama_sampler_chain_add(sampler, llama_sampler_init_dist(UInt32.max)) // random seed self.sampler = sampler! } deinit { llama_sampler_free(sampler) llama_free(ctx) llama_model_free(model) llama_backend_free() } /// Formats one user turn with the LFM2 chat template and streams the reply. func generate(system: String = "You are a helpful assistant.", user: String, maxTokens: Int = 512, onText: (String) -> Void) { llama_memory_clear(llama_get_memory(ctx), true) // start a fresh conversation let prompt = """ <|im_start|>system \(system)<|im_end|> <|im_start|>user \(user)<|im_end|> <|im_start|>assistant """ var tokens = tokenize(prompt, addBOS: true) // addBOS prepends <|startoftext|> var pending: [UInt8] = [] var rc = tokens.withUnsafeMutableBufferPointer { buf in llama_decode(ctx, llama_batch_get_one(buf.baseAddress, Int32(buf.count))) } for _ in 0.. pending += piece(next) if let text = String(bytes: pending, encoding: .utf8) { // wait for complete UTF-8 sequences onText(text) pending.removeAll() } rc = withUnsafeMutablePointer(to: &next) { p in llama_decode(ctx, llama_batch_get_one(p, 1)) } } } private func tokenize(_ text: String, addBOS: Bool) -> [llama_token] { let byteCount = Int32(text.utf8.count) var tokens = [llama_token](repeating: 0, count: Int(byteCount) + 2) let n = llama_tokenize(vocab, text, byteCount, &tokens, Int32(tokens.count), addBOS, true) return n < 0 ? [] : Array(tokens.prefix(Int(n))) } /// Raw UTF-8 bytes for a token; special/control tokens are filtered out (`special: false`). private func piece(_ token: llama_token) -> [UInt8] { var buf = [CChar](repeating: 0, count: 256) let n = llama_token_to_piece(vocab, token, &buf, Int32(buf.count), 0, false) return n <= 0 ? [] : buf.prefix(Int(n)).map { UInt8(bitPattern: $0) } } } ``` Usage from a view model: ```swift theme={"theme":{"light":"github-light","dark":"github-dark"}} let runner = try LFMRunner(modelPath: modelURL.path) Task.detached { runner.generate(user: "What is machine learning?") { text in Task { @MainActor in self.output += text } } } ``` With the upstream `examples/llama.android` `lib` module: ```kotlin theme={"theme":{"light":"github-light","dark":"github-dark"}} import com.arm.aichat.AiChat import java.io.File val engine = AiChat.getInferenceEngine(applicationContext) lifecycleScope.launch { engine.loadModel(File(filesDir, "LFM2.5-1.2B-Instruct-Q4_0.gguf").absolutePath) engine.setSystemPrompt("You are a helpful assistant.") engine.sendUserPrompt("What is machine learning?", predictLength = 512) .collect { piece -> appendToUi(piece) } // Flow of generated text } ``` `InferenceEngine` applies the model's chat template, manages the KV cache across turns, and exposes a `state: StateFlow` you can bind to your UI (`LoadingModel`, `ModelReady`, `Generating`, …). Call `engine.cleanUp()` to reset the conversation and `engine.destroy()` when you are done. The binding's sampler is configured in `lib/src/main/cpp/ai_chat.cpp` (`new_sampler`, default temperature 0.3). Set it to the values for your model — for LFM2.5-1.2B-Instruct: `temp = 0.1`, `top_k = 50`, `penalty_repeat = 1.05` — or expose those fields through the JNI layer. If you write your own JNI wrapper, the C++ side is the same sequence as the Swift example: `llama_model_load_from_file` → `llama_init_from_model` → sampler chain → `llama_tokenize` → `llama_decode` / `llama_sampler_sample` loop. [`examples/simple/simple.cpp`](https://github.com/ggml-org/llama.cpp/blob/master/examples/simple/simple.cpp) and [`examples/simple-chat/simple-chat.cpp`](https://github.com/ggml-org/llama.cpp/blob/master/examples/simple-chat/simple-chat.cpp) are the canonical reference implementations. For multi-turn conversations, prompt-cache reuse, and how to keep the KV cache aligned with the chat template, see [Chat & Streaming](/deployment/on-device/llama-cpp/chat#native-c-api). ## 4. Tune for mobile * **Memory.** Weights are memory-mapped by default (`use_mmap = true`), so they count as file-backed pages rather than app RSS — iOS jetsam and Android LMK treat them far more leniently. Keep `n_ctx` as small as the use case allows; the KV cache scales linearly with it. * **Threads.** Use the performance cores only: `n_threads = activeProcessorCount - 2` is a good starting point. More threads than physical big cores usually slows decoding. * **GPU.** On Apple devices set `n_gpu_layers = 99` to run on Metal. On Android, CPU is the safe default; Vulkan and OpenCL (Adreno) backends exist but need device-specific testing. * **Quantization.** `Q4_0` for the smallest footprint and fastest Arm kernels; `Q4_K_M` when quality matters more than a few hundred MB. * **Vision and audio.** The XCFramework includes `mtmd`; on Android enable `LLAMA_BUILD_MTMD`. See [Vision & Audio](/deployment/on-device/llama-cpp/multimodal). * **Benchmark on hardware.** `llama-bench -m model.gguf -p 512 -n 128` from the prebuilt Android or macOS binaries gives prefill/decode tokens-per-second before you write any app code. See [Hardware Evaluation](/guides/hardware-evaluation). ## Next steps Multi-turn conversations, sampling parameters, prompt caching. Constrain generation to a JSON schema or GBNF grammar. Tool use with LFM2.5's native tool-call parser. Run LFM2.5-VL and LFM2.5-Audio on llama.cpp. # LM Studio Source: https://docs.liquid.ai/deployment/on-device/lm-studio LM Studio is a desktop application for running LLMs locally with a graphical interface. Use LM Studio for local inference with a graphical interface, easy model discovery and download, and quick testing without command-line setup. ## Installation Download and install LM Studio directly from [lmstudio.ai](https://lmstudio.ai/download). ## Downloading Models 1. Open LM Studio and click the **Search** tab (🔍) 2. Search for "LiquidAI" or "LFM2" 3. Select a model and quantization level (`Q4_K_M` recommended) 4. Click **Download** See the [Models page](/lfm/models/complete-library) for all available GGUF models. ## Using the Chat Interface 1. Go to the **Chat** tab (💬) 2. Select your model from the dropdown 3. Adjust parameters (`temperature`, `max_tokens`, `top_p`) in the sidebar 4. Start chatting ## Generation Parameters Control text generation behavior using the GUI sidebar or API parameters. Key parameters: * **`temperature`** (`float`, default 1.0): Controls randomness (0.0 = deterministic, higher = more random). Typical range: 0.1-2.0 * **`top_p`** (`float`, default 1.0): Nucleus sampling - limits to tokens with cumulative probability ≤ top\_p. Typical range: 0.1-1.0 * **`top_k`** (`int`, default 40): Limits to top-k most probable tokens. Typical range: 1-100 * **`max_tokens`** (`int`): Maximum number of tokens to generate * **`repeat_penalty`** (`float`, default 1.0): Penalty for repeating tokens (>1.0 = discourage repetition). Typical range: 1.0-1.5 * **`stop`** (`str` or `list[str]`): Strings that terminate generation when encountered Via the OpenAI-compatible API: ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} response = client.chat.completions.create( model="local-model", messages=[{"role": "user", "content": "What is machine learning?"}], temperature=0.1, max_tokens=512, extra_body={"top_k": 50, "repeat_penalty": 1.05}, ) ``` ## Running the Server Start an OpenAI-compatible server for programmatic access: 1. Go to the **Developer** tab (⚙️) 2. Select your model 3. Click **Start Server** (runs at `http://localhost:1234`) Use the OpenAI Python client: ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} from openai import OpenAI client = OpenAI( base_url="http://localhost:1234/v1", api_key="not-needed" ) response = client.chat.completions.create( model="local-model", # Any string works messages=[ {"role": "user", "content": "What is machine learning?"} ], temperature=0.1, max_tokens=512, extra_body={"top_k": 50, "repeat_penalty": 1.05}, ) print(response.choices[0].message.content) ``` ### Streaming Responses ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} stream = client.chat.completions.create( model="local-model", messages=[ {"role": "user", "content": "Tell me a story."} ], stream=True ) for chunk in stream: if chunk.choices[0].delta.content is not None: print(chunk.choices[0].delta.content, end="") ``` ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} curl http://localhost:1234/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "local-model", "messages": [{"role": "user", "content": "Hello!"}], "temperature": 0.1, "top_k": 50, "repeat_penalty": 1.05 }' ``` ## Vision Models Search for "LiquidAI LFM2-VL" to download vision models. In the **Chat** tab: * Drag and drop images into the chat * Click the image icon to upload * Provide image URLs ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} from openai import OpenAI import base64 client = OpenAI( base_url="http://localhost:1234/v1", api_key="not-needed" ) # Encode image to base64 with open("image.jpg", "rb") as image_file: image_data = base64.b64encode(image_file.read()).decode("utf-8") response = client.chat.completions.create( model="local-model", messages=[ { "role": "user", "content": [ {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_data}"}}, {"type": "text", "text": "What's in this image?"} ] } ] ) print(response.choices[0].message.content) ``` ## Tips * **GPU Acceleration**: Automatically detects and uses available GPUs * **Model Management**: Delete models from the **My Models** section * **Performance**: Adjust GPU layers in server settings for speed/memory balance * **Quantization**: Q4 is faster, Q6/Q8 have better quality # MLX Source: https://docs.liquid.ai/deployment/on-device/mlx MLX is Apple's machine learning framework optimized for Apple Silicon. It provides efficient inference on Mac devices with M-series chips (M1, M2, M3, M4) using Metal acceleration for GPU computing. Use MLX for running models on Apple Silicon Macs with Metal GPU acceleration. MLX leverages unified memory architecture on Apple Silicon, allowing seamless data sharing between CPU and GPU. The `mlx-lm` package provides a simple interface for loading and serving LLMs. ## Installation Install the MLX language model package: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} pip install mlx-lm ``` ## Basic Usage The `mlx-lm` package provides a simple interface for text generation with MLX models. See the [Models page](/lfm/models/complete-library) for all available MLX models, or browse MLX community models at [mlx-community LFM2 models](https://huggingface.co/models?sort=created\&search=mlx-communityLFM2). ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} from mlx_lm import load, generate # Load model and tokenizer model, tokenizer = load("mlx-community/LFM2-1.2B-8bit") # Generate text prompt = "What is machine learning?" # Apply chat template messages = [{"role": "user", "content": prompt}] prompt = tokenizer.apply_chat_template( messages, tokenizer=False, add_generation_prompt=True ) response = generate(model, tokenizer, prompt=prompt, verbose=True) print(response) ``` ### Generation Parameters Control text generation behavior using parameters in the `generate()` function. Key parameters: * **`temperature`** (`float`, default 1.0): Controls randomness (0.0 = deterministic, higher = more random). Typical range: 0.1-2.0 * **`top_p`** (`float`, default 1.0): Nucleus sampling - limits to tokens with cumulative probability ≤ top\_p. Typical range: 0.1-1.0 * **`top_k`** (`int`, default 50): Limits to top-k most probable tokens. Typical range: 1-100 * **`max_tokens`** (`int`): Maximum number of tokens to generate * **`repetition_penalty`** (`float`, default 1.0): Penalty for repeating tokens (>1.0 = discourage repetition). Typical range: 1.0-1.5 Example with custom parameters: ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} response = generate( model, tokenizer, prompt=prompt, temperature=0.3, min_p=0.15, repetition_penalty=1.05, max_tokens=512 ) ``` ## Streaming Generation Stream responses with `stream_generate()`: ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} from mlx_lm import load, stream_generate model, tokenizer = load("mlx-community/LFM2-1.2B-8bit") messages = [{"role": "user", "content": "Tell me a story about space exploration."}] prompt = tokenizer.apply_chat_template( messages, tokenizer=False, add_generation_prompt=True ) for token in stream_generate(model, tokenizer, prompt=prompt, max_tokens=512): print(token, end="", flush=True) ``` ## Serving with mlx-lm MLX can serve models through an OpenAI-compatible API. Start a server with: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} mlx_lm.server --model mlx-community/LFM2-1.2B-8bit --port 8080 ``` ### Using the Server Once running, use the OpenAI Python client: ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} from openai import OpenAI client = OpenAI( base_url="http://localhost:8080/v1", api_key="not-needed" ) response = client.chat.completions.create( model="mlx-community/LFM2-1.2B-8bit", messages=[ {"role": "user", "content": "Explain quantum computing."} ], temperature=0.3, max_tokens=512, extra_body={"min_p": 0.15, "repetition_penalty": 1.05}, ) print(response.choices[0].message.content) ``` You can also use curl to interact with the server: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} curl http://localhost:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "mlx-community/LFM2-1.2B-8bit", "messages": [{"role": "user", "content": "Hello!"}], "temperature": 0.3, "min_p": 0.15, "repetition_penalty": 1.05 }' ``` ## Vision Models LFM2-VL models support both text and image inputs for multimodal inference. Use `mlx_vlm` to load and generate with vision models: ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} from mlx_vlm import load, generate from mlx_vlm.prompt_utils import apply_chat_template from mlx_vlm.utils import load_image_processor from PIL import Image # Load vision model model, processor = load("mlx-community/LFM2-VL-1.6B-8bit") # Load image image = Image.open("path/to/image.jpg") # Create prompt messages = [ { "role": "user", "content": [ {"type": "image"}, {"type": "text", "text": "What's in this image?"} ] } ] # Apply chat template prompt = apply_chat_template(processor, messages) # Generate output = generate(model, processor, image, prompt, verbose=False) print(output) ``` ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} images = [ Image.open("path/to/first.jpg"), Image.open("path/to/second.jpg") ] messages = [ { "role": "user", "content": [ {"type": "image"}, {"type": "image"}, {"type": "text", "text": "What are the differences between these images?"} ] } ] prompt = apply_chat_template(processor, messages) output = generate(model, processor, images, prompt, verbose=False) print(output) ``` # Ollama Source: https://docs.liquid.ai/deployment/on-device/ollama Ollama is a command-line tool for running LLMs locally with a simple interface. It provides easy model management and serving with an OpenAI-compatible API. Use Ollama for quick local model serving with a simple CLI or Docker-based deployment. Ollama uses GGUF models and supports GPU acceleration (CUDA, Metal, ROCm). The official Ollama v0.17.0 (latest stable) from [ollama.com](https://ollama.com) fails with a `missing tensor 'output_norm.weight'` error on the `lfm2moe` architecture. This affects all LFM MoE models (e.g. LFM2-24B-A2B, LFM2-8A-A1B). To run any LFM MoE model you specifically need [v0.17.1-rc0](https://github.com/ollama/ollama/releases/tag/v0.17.1-rc0) or later. ## Installation Download directly from [ollama.com/download](https://ollama.com/download). ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} curl -fsSL https://ollama.com/install.sh | sh ``` Run Ollama with GPU acceleration inside Docker containers: **CPU only:** ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama ``` **NVIDIA GPU:** ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama ``` Then run a model: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} docker exec -it ollama ollama run hf.co/LiquidAI/LFM2.5-1.2B-Instruct-GGUF ``` See the [Ollama Docker documentation](https://ollama.com/blog/ollama-is-now-available-as-an-official-docker-image) for more details. ## Using LFM2 Models Ollama can load GGUF models directly from Hugging Face or from local files. ### Running GGUFs You can run LFM2 models directly from Hugging Face: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} ollama run hf.co/LiquidAI/LFM2.5-1.2B-Instruct-GGUF ``` See the [Models page](/lfm/models/complete-library) for all available GGUF repositories. To use a local GGUF file, first download a model from Hugging Face: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} uv pip install huggingface-hub hf download LiquidAI/LFM2.5-1.2B-Instruct-GGUF {quantization}.gguf --local-dir . ``` Replace `{quantization}` with your preferred quantization level (e.g., `q4_k_m`, `q8_0`). Then run the local model: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} ollama run /path/to/model.gguf ``` For custom configurations (specific quantization, chat template, or parameters), create a Modelfile. Create a plain text file named `Modelfile` (no extension) with the following content: ``` FROM /path/to/model.gguf TEMPLATE """<|startoftext|><|im_start|>system {{ .System }}<|im_end|> <|im_start|>user {{ .Prompt }}<|im_end|> <|im_start|>assistant """ PARAMETER temperature 0.1 PARAMETER top_k 50 PARAMETER repeat_penalty 1.05 PARAMETER stop "<|im_end|>" PARAMETER stop "<|endoftext|>" ``` Import the model with the Modelfile: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} ollama create my-model -f Modelfile ``` Then run it: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} ollama run my-model ``` ## Basic Usage Interact with models through the command-line interface. ### Interactive Chat ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} ollama run hf.co/LiquidAI/LFM2.5-1.2B-Instruct-GGUF ``` Type your messages and press Enter. Use `/bye` to exit. ### Single Prompt ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} ollama run hf.co/LiquidAI/LFM2.5-1.2B-Instruct-GGUF "What is machine learning?" ``` If you imported a model with a custom name using a Modelfile, use that name instead (e.g., `ollama run my-model`). ## Serving Models Ollama automatically starts a server on `http://localhost:11434` with an OpenAI-compatible API for programmatic access. ### Python Client ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} from openai import OpenAI client = OpenAI( base_url="http://localhost:11434/v1", api_key="not-needed" ) response = client.chat.completions.create( model="hf.co/LiquidAI/LFM2.5-1.2B-Instruct-GGUF", messages=[ {"role": "user", "content": "Explain quantum computing."} ], temperature=0.1, extra_body={"top_k": 50, "repeat_penalty": 1.05}, ) print(response.choices[0].message.content) ``` Ollama provides two native API endpoints: **Generate API** (simple completion): ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} curl http://localhost:11434/api/generate -d '{ "model": "hf.co/LiquidAI/LFM2-1.2B-GGUF", "prompt": "What is artificial intelligence?" }' ``` **Chat API** (conversational format): ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} curl http://localhost:11434/api/chat -d '{ "model": "hf.co/LiquidAI/LFM2-1.2B-GGUF", "messages": [ {"role": "user", "content": "Hello!"} ] }' ``` ## Vision Models LFM2-VL GGUF models can also be used for multimodal inference with Ollama. Run a vision model directly and provide images in the chat: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} ollama run hf.co/LiquidAI/LFM2.5-VL-1.6B-GGUF ``` In the interactive chat, you can ask questions about images using the `/image` command followed by the file path: ``` >>> /image path/to/image.jpg What's in this image? ``` Or provide the image path directly in your prompt: ``` >>> Describe the contents of ~/Downloads/photo.png ``` ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} from openai import OpenAI import base64 client = OpenAI( base_url="http://localhost:11434/v1", api_key="not-needed" ) # Encode image to base64 with open("image.jpg", "rb") as image_file: image_data = base64.b64encode(image_file.read()).decode("utf-8") response = client.chat.completions.create( model="hf.co/LiquidAI/LFM2.5-VL-1.6B-GGUF", messages=[ { "role": "user", "content": [ {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_data}"}}, {"type": "text", "text": "What's in this image?"} ] } ] ) print(response.choices[0].message.content) ``` ## Model Management List installed models: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} ollama list ``` Remove a model: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} ollama rm hf.co/LiquidAI/LFM2.5-1.2B-Instruct-GGUF ``` Show model information: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} ollama show hf.co/LiquidAI/LFM2.5-1.2B-Instruct-GGUF ``` # ONNX Source: https://docs.liquid.ai/deployment/on-device/onnx ONNX provides a platform-agnostic inference specification that allows running the model on device-specific runtimes that include CPU, GPU, NPU, and WebGPU. Use ONNX for cross-platform deployment, edge devices, and browser-based inference with WebGPU and Transformers.js. ONNX (Open Neural Network Exchange) is a portable format that enables LFM inference across diverse hardware and runtimes. ONNX models run on CPUs, GPUs, NPUs, and in browsers via WebGPU—making them ideal for edge deployment and web applications. ## LiquidONNX [LiquidONNX](https://github.com/Liquid4All/onnx-export) is the official tool for exporting LFM models to ONNX and running inference. ### Installation ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} git clone https://github.com/Liquid4All/onnx-export.git cd onnx-export uv sync # For GPU inference uv sync --extra gpu ``` ### Supported Models | Family | Quantization Formats | | --------------------------- | --------------------- | | LFM2.5, LFM2 (text) | fp32, fp16, q4, q8 | | LFM2.5-VL, LFM2-VL (vision) | fp32, fp16, q4, q8 | | LFM2-MoE | fp32, fp16, q4, q4f16 | | LFM2.5-Audio | fp32, fp16, q4, q8 | ### Export ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} # Text models - export with all precisions (fp16, q4, q8) uv run lfm2-export LiquidAI/LFM2.5-1.2B-Instruct --precision # Vision-language models uv run lfm2-vl-export LiquidAI/LFM2.5-VL-1.6B --precision # MoE models uv run lfm2-moe-export LiquidAI/LFM2-8B-A1B --precision # Audio models uv run lfm2-audio-export LiquidAI/LFM2.5-Audio-1.5B --precision ``` ### Inference ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} # Text model chat uv run lfm2-infer --model ./exports/LFM2.5-1.2B-Instruct-ONNX/onnx/model_q4.onnx # Vision-language with images uv run lfm2-vl-infer --model ./exports/LFM2.5-VL-1.6B-ONNX \ --images photo.jpg --prompt "Describe this image" # Audio transcription (ASR) uv run lfm2-audio-infer LFM2.5-Audio-1.5B-ONNX --mode asr \ --audio input.wav --precision q4 # Text-to-speech (TTS) uv run lfm2-audio-infer LFM2.5-Audio-1.5B-ONNX --mode tts \ --prompt "Hello, how are you?" --output speech.wav --precision q4 ``` For complete documentation and advanced options, see the [LiquidONNX GitHub repository](https://github.com/Liquid4All/onnx-export). ## Pre-exported Models Many LFM models are available as pre-exported ONNX packages from [LiquidAI](https://huggingface.co/LiquidAI/models?search=onnx) and the [onnx-community](https://huggingface.co/onnx-community). Check the [Model Library](/lfm/models/complete-library) for a complete list of available formats. ### Quantization Options Each ONNX export includes multiple precision levels. **Q4** is recommended for most deployments and supports WebGPU, CPU, and GPU. **FP16** offers higher quality and works on WebGPU and GPU. **Q8** provides a quality/size balance but is server-only (CPU/GPU). **FP32** is the full precision baseline. ## Hugging Face Spaces These are fully deployed examples of WebGPU and ONNX inference with LFM models. Run LFM2 text models directly in your browser with WebGPU acceleration. Speech-to-text and text-to-speech with LFM2.5 Audio in the browser. Vision-language inference with LFM2.5-VL in the browser. ## WebGPU Inference ONNX models run in browsers via [Transformers.js](https://huggingface.co/docs/transformers.js) with WebGPU acceleration. This enables fully client-side inference without server infrastructure. ### Setup 1. Install Transformers.js: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} npm install @huggingface/transformers ``` 2. Enable WebGPU in your browser: * **Chrome/Edge**: Navigate to `chrome://flags/#enable-unsafe-webgpu`, enable, and restart * **Verify**: Check `chrome://gpu` for WebGPU status ### Usage ```javascript theme={"theme":{"light":"github-light","dark":"github-dark"}} import { AutoModelForCausalLM, AutoTokenizer, TextStreamer } from "@huggingface/transformers"; const modelId = "LiquidAI/LFM2.5-1.2B-Instruct-ONNX"; // Load model with WebGPU const tokenizer = await AutoTokenizer.from_pretrained(modelId); const model = await AutoModelForCausalLM.from_pretrained(modelId, { device: "webgpu", dtype: "q4", // or "fp16" }); // Generate with streaming const messages = [{ role: "user", content: "What is the capital of France?" }]; const input = tokenizer.apply_chat_template(messages, { add_generation_prompt: true, return_dict: true, }); const streamer = new TextStreamer(tokenizer, { skip_prompt: true }); const output = await model.generate({ ...input, max_new_tokens: 256, do_sample: false, streamer, }); console.log(tokenizer.decode(output[0], { skip_special_tokens: true })); ``` WebGPU supports Q4 and FP16 precision. Q8 quantization is not available in browser environments. ## Python Inference Install with pip: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} pip install onnxruntime transformers numpy huggingface_hub jinja2 # For GPU support pip install onnxruntime-gpu transformers numpy huggingface_hub jinja2 ``` ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} import numpy as np import onnxruntime as ort from huggingface_hub import hf_hub_download, list_repo_files from transformers import AutoTokenizer # Download Q4 model (recommended) model_id = "LiquidAI/LFM2.5-1.2B-Instruct-ONNX" model_path = hf_hub_download(model_id, "onnx/model_q4.onnx") # Download external data files for f in list_repo_files(model_id): if f.startswith("onnx/model_q4.onnx_data"): hf_hub_download(model_id, f) # Load model and tokenizer session = ort.InferenceSession(model_path) tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True) # Prepare input messages = [{"role": "user", "content": "What is the capital of France?"}] prompt = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True ) inputs = tokenizer.encode(prompt, add_special_tokens=False) input_ids = np.array([inputs], dtype=np.int64) # Initialize KV cache DTYPE_MAP = { "tensor(float)": np.float32, "tensor(float16)": np.float16, "tensor(int64)": np.int64 } cache = {} for inp in session.get_inputs(): if inp.name in {"input_ids", "attention_mask", "position_ids"}: continue shape = [d if isinstance(d, int) else 1 for d in inp.shape] for i, d in enumerate(inp.shape): if isinstance(d, str) and "sequence" in d.lower(): shape[i] = 0 dtype = DTYPE_MAP.get(inp.type, np.float32) cache[inp.name] = np.zeros(shape, dtype=dtype) # Generate tokens seq_len = input_ids.shape[1] generated = [] input_names = {inp.name for inp in session.get_inputs()} for step in range(100): if step == 0: ids = input_ids pos = np.arange(seq_len, dtype=np.int64).reshape(1, -1) else: ids = np.array([[generated[-1]]], dtype=np.int64) pos = np.array([[seq_len + len(generated) - 1]], dtype=np.int64) attn_mask = np.ones((1, seq_len + len(generated)), dtype=np.int64) feed = {"input_ids": ids, "attention_mask": attn_mask, **cache} if "position_ids" in input_names: feed["position_ids"] = pos outputs = session.run(None, feed) next_token = int(np.argmax(outputs[0][0, -1])) generated.append(next_token) # Update cache for i, out in enumerate(session.get_outputs()[1:], 1): name = out.name.replace("present_conv", "past_conv") name = name.replace("present.", "past_key_values.") if name in cache: cache[name] = outputs[i] if next_token == tokenizer.eos_token_id: break print(tokenizer.decode(generated, skip_special_tokens=True)) ``` # Run local agents with LFMs Source: https://docs.liquid.ai/examples/agent-harnesses Run local agents with LFMs by connecting a locally served model to agent harnesses like Hermes Agent, OpenClaw, and Pi. This guide shows how to run an agent harness fully locally with an LFM. The pattern is the same for every harness: they all talk to an OpenAI-compatible endpoint, so you serve the model once and then point your agent harness of choice, such as [Hermes Agent](https://hermes-agent.nousresearch.com), [OpenClaw](https://openclaw.ai), and [Pi](https://pi.dev), at it. ## Serve the model locally Any server that exposes an OpenAI-compatible `/v1` endpoint works. Install one backend and start it with tool calling enabled. Each backend serves on its own default port, so note the local URL yours prints. You point your harness at that URL. Each backend uses its own default port, so your endpoint depends on which one you run. llama.cpp and MLX use `8080`, vLLM uses `8000`, SGLang uses `30000`, LM Studio uses `1234`, and Atomic Chat uses `1337`. The examples in this guide use `http://localhost:8080/v1`. When you configure a harness, replace the port with your server's. ### Model configuration [LFM2.5-2.6B](/lfm/models/lfm25-2.6b) is a dense 2.6B-parameter model built for on-device deployment. It runs fast on consumer hardware and supports tool calling, which makes it a good fit for agentic workloads. The two settings worth choosing up front are the quantization and the context length. Both trade memory for quality or capacity, so pick them to fit your hardware. **Quantization.** Because LFM2.5-2.6B is small, you have room to trade size for quality. For the GGUF quants, which cover llama.cpp and LM Studio, we recommend starting with `Q4_K_M` and stepping up to `Q8_0` or `BF16` depending on your available memory. | Quant | Size | Notes | | -------- | ------- | ----------------------------------------------------------- | | `Q4_K_M` | 1.67 GB | Best balance of size and quality (recommended) | | `Q6_K` | 2.22 GB | Better quality | | `Q8_0` | 2.87 GB | Near-lossless and a safe choice for tool-heavy agentic work | | `BF16` | 5.4 GB | Full precision for maximum fidelity and benchmarking | MLX uses its own quantization. Pick the 4-bit, 6-bit, 8-bit, or bf16 build from the MLX repo. vLLM and SGLang run the full-precision weights on GPU. **Context length**. Agents consume context quickly. If you hit truncation or context-overflow errors mid-run, raise the served context or trim the agent's history. LFM2.5-2.6B supports up to 128K tokens. The examples serve the full window, but if you're memory-constrained, serve a smaller window such as 32K tokens, which is usually plenty for a single agent task. ### Start a server Install one backend and start it with tool calling enabled. **Install:** ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} brew install llama.cpp # macOS winget install llama.cpp # Windows ``` For Linux and build-from-source options, see the [llama.cpp guide](/deployment/on-device/llama-cpp). **Run:** The `-hf` flag auto-downloads the GGUF. ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} llama-server -hf LiquidAI/LFM2.5-2.6B-GGUF:Q4_K_M \ --jinja \ --port 8080 \ -c 131072 \ -fa on \ -ngl 99 \ --temp 0.1 \ --top-k 50 \ --repeat-penalty 1.1 ``` | Flag | Meaning | | ----------- | ------------------------------------------------- | | `--jinja` | **Enables tool calling** via the model's template | | `-c 131072` | Context window (128K) | | `-fa on` | Flash attention (needs a Metal or CUDA build) | | `-ngl 99` | Offload all layers to GPU | **Install:** Download and install [LM Studio](https://lmstudio.ai), then search for **LFM2.5-2.6B** in the model catalog and download the `Q4_K_M` GGUF. See the [LM Studio guide](/deployment/on-device/lm-studio). **Run:** Open the **Developer / Local Server** tab, then: 1. Load the **LFM2.5-2.6B** model. 2. Enable **tool use** in the model settings. 3. Set the context length in the model settings. 4. Click **Start Server**. It serves at `http://localhost:1234`. **Install:** Download and install [Atomic Chat](https://atomic.chat), then search for **LFM2.5-2.6B** in the model library and download the `Q4_K_M` GGUF. See the [Atomic Chat guide](/deployment/on-device/atomic-chat). **Run:** 1. Load **LFM2.5-2.6B** in a chat. 2. Open the **Integrations** tab. 3. Click **Start Server**. It serves the loaded model at `http://localhost:1337` by default; use the port shown in the app. The same Integrations screen also includes one-click agent launchers, so you can run a harness right next to the server. **Install** (Apple Silicon only): ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} pip install mlx-lm ``` See the [MLX guide](/deployment/on-device/mlx). **Run:** `mlx_lm.server` exposes an OpenAI-compatible endpoint: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} mlx_lm.server --model LiquidAI/LFM2.5-2.6B-MLX --port 8080 ``` Confirm your `mlx-lm` version forwards tools to the chat template. **Install:** ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} pip install vllm ``` For GPU servers rather than laptops. See the [vLLM guide](/deployment/gpu-inference/vllm). **Run:** Tool calling requires explicit flags: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} vllm serve LiquidAI/LFM2.5-2.6B \ --enable-auto-tool-choice \ --tool-call-parser lfm2 ``` Serves at `http://localhost:8000/v1`. **Install:** ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} uv pip install "sglang>=0.5.10" ``` For GPU servers rather than laptops. See the [SGLang guide](/deployment/gpu-inference/sglang). **Run:** Tool calling requires an explicit parser flag: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} sglang serve \ --model-path LiquidAI/LFM2.5-2.6B \ --host 0.0.0.0 \ --port 30000 \ --tool-call-parser lfm2 ``` Serves at `http://localhost:30000/v1`. Check that the model is loaded and reachable (replace `8080` with your server's port): ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} curl http://localhost:8080/v1/models ``` ## Connect your agent harness Every harness connects the same way: install it, point it at your local server, then run. The examples below use `http://localhost:8080/v1` and model id `LFM2.5-2.6B`. Replace the port with the one your server prints. Only the exact commands differ per harness. Docs: [Custom / self-hosted providers](https://hermes-agent.nousresearch.com/docs/integrations/providers#custom--self-hosted-llm-providers). **Install:** ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup ``` **Configure:** Use the interactive wizard: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} hermes model # choose "Custom endpoint (self-hosted / vLLM / etc.)" # API base URL: http://localhost:8080/v1 # API key: (leave empty for local) # Model name: LFM2.5-2.6B ``` Or set it directly, then **enable tool-use enforcement** (without it, the model tends to *describe* actions instead of calling tools): ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} hermes config set model.provider custom hermes config set model.base_url http://localhost:8080/v1 hermes config set model.default LFM2.5-2.6B hermes config set model.context_length 131072 hermes config set model.api_mode chat_completions hermes config set agent.tool_use_enforcement true ``` **Run:** ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} hermes ``` > \[!Note] > If `web_search` is missing from the model's available tools, it may be due to `search` or `browser` being listed in `agent.disabled_toolsets`. Remove both entries in `hermes config edit` and restart Hermes. Docs: [Getting started](https://docs.openclaw.ai/start/getting-started) and [Local models](https://docs.openclaw.ai/gateway/local-models). **Install:** ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} curl -fsSL https://openclaw.ai/install.sh | bash # macOS / Linux openclaw onboard --install-daemon ``` **Configure:** Add a custom OpenAI-compatible provider (JSON5) under `models.providers`. Tool calling is on by default for custom providers. ```json5 theme={"theme":{"light":"github-light","dark":"github-dark"}} { models: { mode: "merge", providers: { local: { baseUrl: "http://localhost:8080/v1", apiKey: "sk-local", // a local marker is accepted for loopback api: "openai-completions", models: [ { id: "LFM2.5-2.6B", name: "LFM2.5-2.6B", input: ["text"], contextWindow: 131072, maxTokens: 8192, cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 }, }, ], }, }, }, } ``` Select it as the active model: ```json5 theme={"theme":{"light":"github-light","dark":"github-dark"}} { agents: { defaults: { model: { primary: "local/LFM2.5-2.6B" } } } } ``` **Run:** ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} openclaw dashboard ``` This opens the Control UI in your browser, where you enter your task. Docs: [Pi models documentation](https://pi.dev/docs/latest/models). **Install:** ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} npm install -g --ignore-scripts @earendil-works/pi-coding-agent # recommended # or: curl -fsSL https://pi.dev/install.sh | sh ``` **Configure:** Add the provider to `~/.pi/agent/models.json` (the file reloads when you run `/model`, so no restart is needed): ```json theme={"theme":{"light":"github-light","dark":"github-dark"}} { "providers": { "local": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "local", "models": [{ "id": "LFM2.5-2.6B" }] } } } ``` `apiKey` can be any placeholder for a keyless local server. If Pi flags unsupported features, add a `compat` block, e.g. `"compat": { "supportsReasoningEffort": false }`. **Run:** ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} pi ``` Then select the model with `/model`. Now, you have your agent harness running fully locally on your machine. ## References * [LFM2.5-2.6B](/lfm/models/lfm25-2.6b) * [llama.cpp deployment](/deployment/on-device/llama-cpp) * [vLLM deployment](/deployment/gpu-inference/vllm) * [SGLang deployment](/deployment/gpu-inference/sglang) * [Hermes Agent documentation](https://hermes-agent.nousresearch.com/docs/) * [OpenClaw documentation](https://docs.openclaw.ai/) * [Pi documentation](https://pi.dev/docs/) # Build AI Agents with Koog Framework on Android Source: https://docs.liquid.ai/examples/android/leap-koog-agent This example uses the LEAP SDK, which is **deprecated**. It remains a useful architectural reference, but for new Android apps embed llama.cpp directly — see [iOS & Android](/deployment/on-device/llama-cpp/mobile) and [Migrating from LEAP SDK](/deployment/on-device/llama-cpp/migrating-from-leap-sdk). Browse the complete example on GitHub This example demonstrates how to integrate the **Koog framework** with LeapSDK on Android to build intelligent AI agents. Koog is an open-source framework for creating AI agents that can understand natural language, invoke tools, manage context, and integrate with MCP (Model Context Protocol) servers. LeapKoogAgent shows how to bring autonomous agent capabilities to mobile devices, enabling apps that can reason, plan, and execute complex tasks on behalf of users—all running locally on Android. ## What's inside? LeapKoogAgent showcases advanced AI agent capabilities: * **Koog Framework Integration** - Build structured AI agents with reasoning capabilities * **Natural Language Understanding** - Process user intents and commands * **Tool Invocation** - Enable agents to call functions and use external tools * **Context Management** - Maintain conversation state and agent memory * **Extensible Architecture** - Easily add new capabilities and tools * **MCP Server Integration** - Connect to Model Context Protocol servers for tool retrieval and execution * **Event Handling** - Respond to agent events and state changes * **On-device Agent Runtime** - Complete agent execution without cloud dependency This example demonstrates building production-ready AI agents that run entirely on Android devices. ## What is the Koog Framework? **Koog** is an open-source framework for building AI agents. It provides a structured approach to creating agents that can: * **Reason and plan** - Break down complex tasks into steps * **Use tools** - Invoke functions to accomplish specific tasks (calculations, API calls, database queries) * **Maintain context** - Remember conversation history and user preferences * **Handle events** - React to user input, system events, or external triggers * **Execute workflows** - Chain multiple actions together to complete objectives * **Integrate with MCP** - Access tools and resources from Model Context Protocol servers **Key concepts:** * **Agents** - Autonomous entities that process input, reason, and take actions * **Tools** - Functions that agents can invoke (e.g., search, calculate, retrieve data) * **Context** - The agent's knowledge state including conversation history * **MCP (Model Context Protocol)** - A standard for exposing tools and resources to agents * **Events** - Notifications about agent state changes or actions **Common agent use cases:** * Personal assistants that manage tasks and calendars * Customer service bots with access to knowledge bases * Data analysis agents that query databases and generate reports * Workflow automation agents that orchestrate multiple services * Educational tutors that adapt to learner needs * Smart home controllers that understand natural language commands Explore the Koog framework documentation and examples ## Environment setup Before running this example, ensure you have the following: Download and install [Android Studio](https://developer.android.com/studio) (latest stable version recommended). Make sure you have: * Android SDK installed * An Android device or emulator configured * USB debugging enabled (for physical devices) This example requires: * **Minimum SDK**: API 31 (Android 12) * **Target SDK**: API 36 * **Kotlin**: 2.3.0 or higher **Hardware recommendations:** * At least 4GB RAM (agents require more memory for reasoning) * Sufficient storage for model bundles LeapKoogAgent uses an **LFM2-1.2B** GGUF checkpoint suited for tool use. You can either let `LeapModelDownloader.loadModel(...)` pull it from the [LEAP Model Library](https://leap.liquid.ai/models) on first launch, or push it manually via ADB: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} # Ensure device is connected adb devices # Create directory (world-readable so the app can read it) adb shell mkdir -p /data/local/tmp/liquid/ # Push the GGUF model file adb push lfm2-1.2b-q5_k_m.gguf /data/local/tmp/liquid/ # Verify deployment adb shell ls -lh /data/local/tmp/liquid/ ``` **Note:** Apps cannot read `/tmp/` on Android — use `/data/local/tmp//` for ADB-pushed assets (matches the other Android examples). If you deploy to a different location, update the `modelPath` in your app code accordingly. The example snippets below use `loadSimpleModel(model: ModelSource(...))` to load the sideloaded file; switch to `loadModel(modelName:, quantizationType:)` if you'd rather have the SDK download the model automatically. Add the required dependencies to your app-level `build.gradle.kts`: ```kotlin theme={"theme":{"light":"github-light","dark":"github-dark"}} dependencies { // LeapSDK for on-device AI (0.10.0+) implementation("ai.liquid.leap:leap-sdk:0.10.7") implementation("ai.liquid.leap:leap-model-downloader:0.10.7") // Koog framework for AI agents implementation("ai.koog:koog-agents:0.5.0") // JSON processing implementation("org.jetbrains.kotlinx:kotlinx-serialization-json:1.6.0") // Coroutines for async operations implementation("org.jetbrains.kotlinx:kotlinx-coroutines-android:1.7.3") // Jetpack Compose implementation(platform("androidx.compose:compose-bom:2024.01.00")) implementation("androidx.compose.ui:ui") implementation("androidx.compose.material3:material3") // ViewModel implementation("androidx.lifecycle:lifecycle-viewmodel-compose:2.7.0") } ``` ## How to run it Follow these steps to build and run AI agents on Android: 1. **Clone the repository** ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} git clone https://github.com/Liquid4All/LeapSDK-Examples.git cd LeapSDK-Examples/Android/LeapKoogAgent ``` 2. **Deploy the model** * Follow the ADB commands in the setup section above * Ensure the GGUF is at `/data/local/tmp/liquid/lfm2-1.2b-q5_k_m.gguf` 3. **Open in Android Studio** * Launch Android Studio * Select "Open an existing project" * Navigate to the `LeapKoogAgent` folder 4. **Build the project** * Wait for Gradle sync to complete * Resolve any dependency issues 5. **Run on device or emulator** * Connect your Android device or start an emulator * Click "Run" or press `Shift + F10` 6. **Interact with the agent** * On launch, the agent will initialize (may take 10-20 seconds) * Enter a command or question in the input field * Watch the agent reason, plan, and execute tasks * The agent can invoke tools to accomplish complex objectives * Try commands like: * "Calculate the sum of 25 and 37" * "What's the weather like today?" (if weather tool is configured) * "Set a reminder for 3pm" * "Search for information about quantum computing" ## Understanding the architecture ### Koog Agent Initialization Load the LEAP model first, then bridge it to a Koog agent. The Koog APIs below are illustrative — see the linked GitHub example for the canonical adapter wiring, since `ai.koog:koog-agents` evolves independently from LeapSDK. ```kotlin theme={"theme":{"light":"github-light","dark":"github-dark"}} import ai.liquid.leap.ModelRunner import ai.liquid.leap.manifest.ModelSource import ai.liquid.leap.downloader.LeapModelDownloader class AgentViewModel(application: Application) : AndroidViewModel(application) { private val downloader = LeapModelDownloader(application) private lateinit var runner: ModelRunner private lateinit var agent: KoogAgent fun initializeAgent() { viewModelScope.launch(Dispatchers.Default) { // Sideloaded GGUF that was pushed via ADB (see Model Setup). runner = downloader.loadSimpleModel( model = ModelSource( modelPath = "/data/local/tmp/liquid/lfm2-1.2b-q5_k_m.gguf", modelName = "LFM2-1.2B", quantizationId = "Q5_K_M", ), ) // Bridge the runner to Koog. The adapter type below is example // pseudocode — the canonical wiring lives in the linked GitHub repo. agent = KoogAgent.Builder() .withModel(LeapModelAdapter(runner)) .withTools(getAvailableTools()) .withSystemPrompt(""" You are a helpful AI assistant running on an Android device. You can help users by answering questions and using available tools. Be concise and helpful in your responses. """.trimIndent()) .build() _agentState.value = AgentState.Ready } } } ``` ### Defining Agent Tools Create tools that the agent can invoke: ```kotlin theme={"theme":{"light":"github-light","dark":"github-dark"}} fun getAvailableTools(): List { return listOf( // Calculator tool KoogTool( name = "calculate", description = "Performs mathematical calculations", parameters = mapOf( "expression" to ToolParameter( type = "string", description = "Mathematical expression to evaluate" ) ), handler = { params -> val expression = params["expression"] as String val result = evaluateExpression(expression) ToolResult.success(result.toString()) } ), // Time/date tool KoogTool( name = "get_current_time", description = "Gets the current date and time", parameters = emptyMap(), handler = { val now = System.currentTimeMillis() val dateTime = SimpleDateFormat("yyyy-MM-dd HH:mm:ss", Locale.getDefault()) .format(Date(now)) ToolResult.success(dateTime) } ), // Device info tool KoogTool( name = "get_device_info", description = "Returns information about the Android device", parameters = emptyMap(), handler = { val info = """ Device: ${Build.MANUFACTURER} ${Build.MODEL} Android Version: ${Build.VERSION.RELEASE} SDK Level: ${Build.VERSION.SDK_INT} """.trimIndent() ToolResult.success(info) } ) ) } ``` ### Processing User Input Handle user messages and agent responses: ```kotlin theme={"theme":{"light":"github-light","dark":"github-dark"}} fun sendMessage(userInput: String) { viewModelScope.launch { // Add user message to conversation _messages.add(Message.User(userInput)) // Process with agent _agentState.value = AgentState.Processing try { val response = agent.process(userInput) // Handle agent response when (response) { is AgentResponse.Message -> { _messages.add(Message.Agent(response.content)) } is AgentResponse.ToolCall -> { // Agent decided to use a tool _messages.add(Message.ToolUse( toolName = response.toolName, status = "Executing..." )) // Execute tool and get result val toolResult = agent.executeTool(response) // Agent processes tool result val finalResponse = agent.continueWithToolResult(toolResult) _messages.add(Message.Agent(finalResponse.content)) } is AgentResponse.Error -> { _messages.add(Message.Error(response.message)) } } _agentState.value = AgentState.Ready } catch (e: Exception) { _messages.add(Message.Error("Failed to process: ${e.message}")) _agentState.value = AgentState.Ready } } } ``` ### MCP Server Integration Connect to MCP servers to expand agent capabilities: ```kotlin theme={"theme":{"light":"github-light","dark":"github-dark"}} fun connectToMCPServer(serverUrl: String) { viewModelScope.launch { try { // Connect to MCP server val mcpClient = MCPClient.connect(serverUrl) // Retrieve available tools from server val remoteTols = mcpClient.listTools() // Add remote tools to agent remoteTols.forEach { tool -> agent.addTool(KoogTool( name = tool.name, description = tool.description, parameters = tool.parameters, handler = { params -> // Forward to MCP server mcpClient.executeTool(tool.name, params) } )) } _mcpStatus.value = "Connected to ${remoteTols.size} tools" } catch (e: Exception) { _mcpStatus.value = "Failed to connect: ${e.message}" } } } ``` ### Event Handling Listen to agent events for monitoring and debugging: ```kotlin theme={"theme":{"light":"github-light","dark":"github-dark"}} agent.addEventListener(object : AgentEventListener { override fun onThinking(thought: String) { // Agent is reasoning about the task Log.d("Agent", "Thinking: $thought") } override fun onToolSelected(toolName: String, reasoning: String) { // Agent decided to use a tool Log.d("Agent", "Selected tool: $toolName because: $reasoning") } override fun onToolExecuted(toolName: String, result: ToolResult) { // Tool execution completed Log.d("Agent", "Tool $toolName returned: ${result.output}") } override fun onError(error: AgentError) { // Agent encountered an error Log.e("Agent", "Error: ${error.message}") } }) ``` ### Context Management Maintain conversation context and agent memory: ```kotlin theme={"theme":{"light":"github-light","dark":"github-dark"}} // Save context for later val context = agent.getContext() preferences.edit() .putString("agent_context", Json.encodeToString(context)) .apply() // Restore context in future session val savedContext = preferences.getString("agent_context", null) if (savedContext != null) { val context = Json.decodeFromString(savedContext) agent.restoreContext(context) } ``` ### Resource Cleanup **Important:** Always clean up agent resources properly to prevent memory leaks and ANRs: ```kotlin theme={"theme":{"light":"github-light","dark":"github-dark"}} class AgentViewModel(application: Application) : AndroidViewModel(application) { private lateinit var agent: KoogAgent private lateinit var runner: ModelRunner override fun onCleared() { super.onCleared() // Clean up agent resources try { agent.shutdown() } catch (e: Exception) { Log.e("AgentViewModel", "Error shutting down agent", e) } // Unload the model asynchronously to avoid ANRs. // Do NOT use runBlocking here — it blocks the main thread. CoroutineScope(Dispatchers.IO).launch { try { runner.unload() } catch (e: Exception) { Log.e("AgentViewModel", "Error unloading model", e) } } } } ``` **Why this matters:** * **Avoid `runBlocking`** in `onCleared()` - it blocks the main thread and can cause ANRs (Application Not Responding) * **Use `CoroutineScope(Dispatchers.IO).launch`** instead - model unloading happens asynchronously * **Always clean up agent resources** - prevents memory leaks and ensures proper shutdown * **Catch exceptions** - cleanup should never crash the app ## Results LeapKoogAgent demonstrates powerful autonomous agent capabilities on Android: **Example interaction:** ``` User: "What's 15% of 340?" Agent: [Thinking] I need to calculate 15% of 340. I'll use the calculate tool. Agent: [Using tool: calculate] Expression: "340 * 0.15" Tool Result: 51.0 Agent: The answer is 51. 15% of 340 equals 51. ``` **Multi-step task:** ``` User: "Calculate the average of 10, 20, and 30, then tell me what time it is" Agent: [Thinking] I need to first calculate the average, then get the current time. Agent: [Using tool: calculate] Expression: "(10 + 20 + 30) / 3" Tool Result: 20.0 Agent: [Using tool: get_current_time] Tool Result: 2024-02-05 14:23:17 Agent: The average of 10, 20, and 30 is 20. The current time is 2:23 PM on February 5th, 2024. ``` All agent reasoning and tool execution happens entirely on your Android device. ## Further improvements Here are some ways to extend this example: * **Custom tool library** - Build domain-specific tools for your app * **Multi-agent systems** - Create multiple specialized agents that collaborate * **Persistent memory** - Store long-term memories in a database * **Voice interaction** - Add speech-to-text and text-to-speech * **Proactive agents** - Trigger agents based on time, location, or events * **Knowledge base integration** - Connect to local or remote knowledge sources * **Task scheduling** - Let agents schedule and execute delayed tasks * **Learning capabilities** - Implement feedback loops to improve agent behavior * **Agent marketplace** - Allow users to install pre-built agent configurations * **Multi-modal inputs** - Process images, audio, and text together * **Security and permissions** - Implement fine-grained tool access control * **Agent telemetry** - Track agent performance and decision quality * **Offline-first design** - Ensure agents work without internet connectivity * **Cross-device sync** - Sync agent context across multiple devices ## Need help? Connect with the community and ask questions about this example. # Generate Structured Recipes with Constrained Output Source: https://docs.liquid.ai/examples/android/recipe-generator-constrained-output This example uses the LEAP SDK, which is **deprecated**. It remains a useful architectural reference, but for new Android apps embed llama.cpp directly — see [iOS & Android](/deployment/on-device/llama-cpp/mobile) and [Migrating from LEAP SDK](/deployment/on-device/llama-cpp/migrating-from-leap-sdk). Browse the complete example on GitHub This example demonstrates **constrained generation** with LeapSDK on Android. Instead of generating free-form text, the RecipeGenerator app produces structured JSON output that follows a predefined schema, ensuring consistent and parseable results. Constrained generation is essential for building reliable AI-powered applications where the output needs to integrate with downstream systems or databases. This example shows how to enforce structure while maintaining creative and useful AI-generated content. ## What's inside? The RecipeGenerator demonstrates advanced LeapSDK capabilities: * **Structured Output Generation** - Enforce JSON schema constraints on model outputs * **Automatic Model Downloading** - Models download automatically via `LeapDownloader` on first run * **Constrained Decoding** - Guide the model to produce valid, parseable data structures * **Schema Validation** - Ensure generated recipes follow a specific format * **Type Safety** - Parse AI outputs directly into Kotlin data classes with kotlinx.serialization * **@Generatable Annotation** - Use LeapSDK's annotation for simplified structured output generation * **Production-ready Pattern** - Integrate AI outputs with databases and business logic * **Practical Use Case** - Generate recipes with ingredients, steps, and metadata This pattern is applicable to many use cases beyond recipes: generating product catalogs, form data, API responses, database entries, and more. ## What is constrained generation? **Constrained generation** refers to the ability to enforce specific formatting or structural requirements on model outputs. Instead of generating free-form text, the model produces outputs that conform to a predefined schema or pattern. **Benefits:** * **Reliability** - Guaranteed parseable output every time * **Type Safety** - Direct integration with typed programming languages * **Validation** - Automatic schema compliance * **Downstream Integration** - Feed structured data into databases, APIs, or UIs * **Error Reduction** - Eliminate parsing errors from inconsistent formats **Common use cases:** * JSON API responses * Database records * Form submissions * Product catalogs * Configuration files * Structured reports Read the complete guide in the official Leap documentation ## Environment setup Before running this example, ensure you have the following: Download and install [Android Studio](https://developer.android.com/studio) (latest stable version recommended). Make sure you have: * Android SDK installed * An Android device or emulator configured * USB debugging enabled (for physical devices) This example requires: * **Minimum SDK**: API 31 (Android 12) * **Target SDK**: API 36 * **Kotlin**: 2.3.0 or higher * **LeapSDK**: 0.10.0 or higher * **Internet connectivity**: Required for first-time model download This example uses **LeapSDK 0.10.0+** with automatic model downloading capabilities. **Automatic Model Management** The app uses `LeapModelDownloader` to download and cache **LFM2-1.2B** on first run: * On first launch, the model downloads automatically from the [LEAP Model Library](https://leap.liquid.ai/models) * Models are cached locally for subsequent app launches * No manual ADB push or file transfer required * Internet connectivity is required only for the initial download **First Run Experience:** 1. Launch the app (internet connection required) 2. The model downloads automatically (may take 2-5 minutes depending on connection) 3. Once cached, subsequent launches work offline 4. The model persists across app restarts **Manual Deployment (Alternative)** If you prefer manual deployment or need offline-first installation, push a GGUF file via ADB and load it with `loadSimpleModel(model: ModelSource(...))`: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} # Push the GGUF file to device storage adb push lfm2-1.2b-q5_k_m.gguf /data/local/tmp/liquid/ ``` Add the required dependencies to your app-level `build.gradle.kts`: ```kotlin theme={"theme":{"light":"github-light","dark":"github-dark"}} dependencies { // LeapSDK + the Android downloader module implementation("ai.liquid.leap:leap-sdk:0.10.7") implementation("ai.liquid.leap:leap-model-downloader:0.10.7") // Kotlin serialization for type-safe parsing implementation("org.jetbrains.kotlinx:kotlinx-serialization-json:1.6.0") // Jetpack Compose (if using Compose UI) implementation(platform("androidx.compose:compose-bom:2024.01.00")) implementation("androidx.compose.ui:ui") implementation("androidx.compose.material3:material3") // ViewModel implementation("androidx.lifecycle:lifecycle-viewmodel-ktx:2.7.0") } ``` **Also enable the Kotlin serialization plugin** in your app-level `build.gradle.kts`: ```kotlin theme={"theme":{"light":"github-light","dark":"github-dark"}} plugins { id("org.jetbrains.kotlin.plugin.serialization") version "2.3.20" } ``` ## How to run it Follow these steps to generate structured recipes: 1. **Clone the repository** ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} git clone https://github.com/Liquid4All/LeapSDK-Examples.git cd LeapSDK-Examples/Android/RecipeGenerator ``` 2. **Open in Android Studio** * Launch Android Studio * Select "Open an existing project" * Navigate to the `RecipeGenerator` folder and open it 3. **Gradle sync** * Wait for Gradle to sync all dependencies * Ensure LeapSDK 0.10.7 is downloaded 4. **Run the app** * Connect your Android device or start an emulator * **Ensure internet connectivity** (required for first-time model download) * Click "Run" or press `Shift + F10` * Select your target device 5. **First launch - model download** * On first run, the app will automatically download the LFM2-1.2B model * This may take 2-5 minutes depending on your connection * A loading indicator will show download progress * The model is cached for future use 6. **Generate recipes** * Enter ingredients or a dish name (e.g., "pasta carbonara", "chocolate chip cookies") * Tap "Generate Recipe" * The app will produce a structured recipe with ingredients, steps, and metadata * All output will be valid JSON conforming to the recipe schema **After First Run**: The model is cached locally. Subsequent app launches work offline and start immediately without downloading. ## Code walkthrough The core business logic is implemented in `MainActivityViewModel.kt`. Here's how it works: ### Define the Recipe Schema First, define the data structure you want the AI to generate: ```kotlin theme={"theme":{"light":"github-light","dark":"github-dark"}} @Serializable data class Recipe( val name: String, val description: String, val servings: Int, val prepTime: String, val cookTime: String, val difficulty: String, val ingredients: List, val instructions: List, val tags: List ) @Serializable data class Ingredient( val item: String, val amount: String, val unit: String ) ``` ### Declare the Recipe Schema with `@Generatable` Annotate the data class with `@Generatable`. LeapSDK derives the JSON schema from the Kotlin types and enforces it during generation — no hand-written schema string required. ```kotlin theme={"theme":{"light":"github-light","dark":"github-dark"}} import ai.liquid.leap.structuredoutput.Generatable import ai.liquid.leap.structuredoutput.Guide import kotlinx.serialization.Serializable @Serializable @Generatable("A complete recipe with metadata, ingredients, and instructions.") data class Recipe( val name: String, val description: String, val servings: Int, val prepTime: String, val cookTime: String, @Guide("One of: easy, medium, hard.") val difficulty: String, val ingredients: List, val instructions: List, val tags: List, ) @Serializable @Generatable("A single recipe ingredient with amount and unit.") data class Ingredient( val item: String, val amount: String, val unit: String, ) ``` ### Load the Model and Wire Constrained Generation ```kotlin theme={"theme":{"light":"github-light","dark":"github-dark"}} import ai.liquid.leap.GenerationOptions import ai.liquid.leap.ModelRunner import ai.liquid.leap.message.ChatMessage import ai.liquid.leap.message.MessageResponse import ai.liquid.leap.downloader.LeapModelDownloader import android.app.Application import androidx.lifecycle.AndroidViewModel import androidx.lifecycle.viewModelScope import kotlinx.coroutines.CoroutineScope import kotlinx.coroutines.Dispatchers import kotlinx.coroutines.flow.MutableStateFlow import kotlinx.coroutines.flow.StateFlow import kotlinx.coroutines.flow.asStateFlow import kotlinx.coroutines.flow.onEach import kotlinx.coroutines.launch import kotlinx.serialization.json.Json class MainActivityViewModel(application: Application) : AndroidViewModel(application) { private val downloader = LeapModelDownloader(application) private var runner: ModelRunner? = null private val _downloadProgress = MutableStateFlow(0f) val downloadProgress: StateFlow = _downloadProgress.asStateFlow() private val _recipeState = MutableStateFlow(RecipeState.Idle) val recipeState: StateFlow = _recipeState.asStateFlow() fun initializeModel() { viewModelScope.launch { runner = downloader.loadModel( modelName = "LFM2-1.2B", quantizationType = "Q5_K_M", progress = { pd -> _downloadProgress.value = if (pd.total > 0) pd.bytes.toFloat() / pd.total else 0f }, ) } } } ``` ### Generate Structured Recipes `GenerationOptions.build { setResponseFormatType() }` tells the engine to constrain the stream to the schema derived from `@Generatable`. The streamed `Chunk` values arrive as JSON; concatenate them and decode at the end with `kotlinx-serialization`. ```kotlin theme={"theme":{"light":"github-light","dark":"github-dark"}} fun generateRecipe(userInput: String) { val runner = runner ?: return viewModelScope.launch { _recipeState.value = RecipeState.Loading val prompt = """ Generate a detailed recipe for: $userInput Include the recipe name, description, servings, preparation time, cooking time, difficulty level, complete ingredient list with amounts, step-by-step instructions, and relevant tags. """.trimIndent() val conversation = runner.createConversation() val options = GenerationOptions.build { temperature = 0.3f minP = 0.15f repetitionPenalty = 1.05f setResponseFormatType() } try { val buffer = StringBuilder() conversation.generateResponse(ChatMessage(ChatMessage.Role.USER, prompt), options) .onEach { resp -> if (resp is MessageResponse.Chunk) buffer.append(resp.text) } .collect() val recipe = Json.decodeFromString(buffer.toString()) _recipeState.value = RecipeState.Success(recipe) } catch (e: Exception) { _recipeState.value = RecipeState.Error(e.message ?: "Failed to generate recipe") } } } ``` ### Resource Cleanup **Important:** Always clean up model resources properly in your ViewModel: ```kotlin theme={"theme":{"light":"github-light","dark":"github-dark"}} override fun onCleared() { super.onCleared() val runner = runner ?: return // Unload the model asynchronously to avoid ANRs. // Do NOT use runBlocking here — it blocks the main thread. CoroutineScope(Dispatchers.IO).launch { try { runner.unload() } catch (e: Exception) { Log.e("RecipeViewModel", "Error unloading model", e) } } } ``` **Best practices:** * Never use `runBlocking` in `onCleared()` - it causes ANRs (Application Not Responding) * Use `CoroutineScope(Dispatchers.IO).launch` for async cleanup * Always catch exceptions to prevent crashes during cleanup * This ensures smooth app shutdown without blocking the UI ### Display the Recipe The structured output can be easily rendered in the UI: ```kotlin theme={"theme":{"light":"github-light","dark":"github-dark"}} @Composable fun RecipeDisplay(recipe: Recipe) { Column(modifier = Modifier.padding(16.dp)) { Text(text = recipe.name, style = MaterialTheme.typography.headlineMedium) Text(text = recipe.description, style = MaterialTheme.typography.bodyMedium) Row { Text("Servings: ${recipe.servings}") Spacer(modifier = Modifier.width(16.dp)) Text("Difficulty: ${recipe.difficulty}") } Text("Prep: ${recipe.prepTime} | Cook: ${recipe.cookTime}") Text("Ingredients:", style = MaterialTheme.typography.titleMedium) recipe.ingredients.forEach { ingredient -> Text("• ${ingredient.amount} ${ingredient.unit} ${ingredient.item}") } Text("Instructions:", style = MaterialTheme.typography.titleMedium) recipe.instructions.forEachIndexed { index, step -> Text("${index + 1}. $step") } FlowRow { recipe.tags.forEach { tag -> Chip(text = tag) } } } } ``` ## Results The RecipeGenerator produces consistently formatted recipes: **Example Output:** ```json theme={"theme":{"light":"github-light","dark":"github-dark"}} { "name": "Classic Chocolate Chip Cookies", "description": "Soft and chewy chocolate chip cookies with a perfect golden-brown exterior", "servings": 24, "prepTime": "15 minutes", "cookTime": "12 minutes", "difficulty": "easy", "ingredients": [ {"item": "all-purpose flour", "amount": "2.25", "unit": "cups"}, {"item": "butter", "amount": "1", "unit": "cup"}, {"item": "granulated sugar", "amount": "0.75", "unit": "cup"}, {"item": "brown sugar", "amount": "0.75", "unit": "cup"}, {"item": "eggs", "amount": "2", "unit": "large"}, {"item": "vanilla extract", "amount": "2", "unit": "tsp"}, {"item": "baking soda", "amount": "1", "unit": "tsp"}, {"item": "salt", "amount": "1", "unit": "tsp"}, {"item": "chocolate chips", "amount": "2", "unit": "cups"} ], "instructions": [ "Preheat oven to 375°F (190°C)", "Cream together butter and both sugars until light and fluffy", "Beat in eggs one at a time, then add vanilla extract", "In a separate bowl, whisk together flour, baking soda, and salt", "Gradually blend dry ingredients into butter mixture", "Stir in chocolate chips", "Drop rounded tablespoons of dough onto ungreased cookie sheets", "Bake for 9-11 minutes or until golden brown", "Cool on baking sheet for 2 minutes before transferring to a wire rack" ], "tags": ["dessert", "baking", "cookies", "chocolate", "classic"] } ``` ![RecipeGenerator Screenshot](https://raw.githubusercontent.com/Liquid4All/LeapSDK-Examples/main/Android/RecipeGenerator/docs/screenshot.png) The interface shows the structured recipe data rendered beautifully with proper formatting, making it easy to read and follow. ## Further improvements Here are some ways to extend this example: * **Database integration** - Save generated recipes to Room database * **Multiple cuisines** - Add cuisine type selector (Italian, Mexican, Asian, etc.) * **Dietary restrictions** - Filter for vegan, gluten-free, keto, etc. * **Nutritional information** - Extend schema to include calories, protein, carbs * **Scaling calculator** - Adjust ingredient amounts based on servings * **Shopping list generation** - Extract ingredients into a shareable shopping list * **Image generation** - Integrate with image models to visualize the dish * **Recipe sharing** - Export as PDF or share via social media * **User ratings** - Allow users to rate and review generated recipes * **Variation generator** - Generate recipe variations (e.g., "make it vegan") * **Meal planning** - Combine multiple recipes into weekly meal plans * **Ingredient substitution** - Suggest alternatives for missing ingredients ## Need help? Connect with the community and ask questions about this example. # Product Slogan Generator with LeapSDK Source: https://docs.liquid.ai/examples/android/slogan-generator This example uses the LEAP SDK, which is **deprecated**. It remains a useful architectural reference, but for new Android apps embed llama.cpp directly — see [iOS & Android](/deployment/on-device/llama-cpp/mobile) and [Migrating from LEAP SDK](/deployment/on-device/llama-cpp/migrating-from-leap-sdk). Browse the complete example on GitHub This example demonstrates how to use LeapSDK for **single-turn generation** tasks on Android. The SloganApp generates creative product slogans on-device using local language models, showcasing how to integrate AI capabilities for specific product needs without requiring cloud connectivity. Built with traditional **Android Views**, this example provides a different UI approach compared to Jetpack Compose, making it ideal for teams working with legacy Android codebases. ## What's inside? The SloganApp showcases: * **Single-turn Generation** - Generate creative output without maintaining conversation context * **Traditional Android Views UI** - Implementation using XML layouts and View-based architecture * **On-device Processing** - Complete privacy with local model inference * **Simple Integration** - Minimal setup for focused AI tasks * **Product-focused Prompting** - Optimized prompts for marketing and slogan generation This example is perfect for understanding how to integrate LeapSDK into existing Android applications that use the traditional View system rather than Jetpack Compose. ## Environment setup Before running this example, ensure you have the following: Download and install [Android Studio](https://developer.android.com/studio) (latest stable version recommended). Make sure you have: * Android SDK installed * An Android device or emulator configured * USB debugging enabled (for physical devices) This example requires: * **Minimum SDK**: API 31 (Android 12) * **Target SDK**: API 36 * **Kotlin**: 2.3.0 or higher Add the LeapSDK to your app-level `build.gradle.kts`: ```kotlin theme={"theme":{"light":"github-light","dark":"github-dark"}} dependencies { implementation("ai.liquid.leap:leap-sdk:0.10.7") implementation("ai.liquid.leap:leap-model-downloader:0.10.7") // Android UI components implementation("androidx.appcompat:appcompat:1.6.1") implementation("com.google.android.material:material:1.11.0") implementation("androidx.constraintlayout:constraintlayout:2.1.4") } ``` ## How to run it Follow these steps to generate product slogans: 1. **Clone the repository** ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} git clone https://github.com/Liquid4All/LeapSDK-Examples.git cd LeapSDK-Examples/Android/SloganApp ``` 2. **Open in Android Studio** * Launch Android Studio * Select "Open an existing project" * Navigate to the `SloganApp` folder and open it 3. **Gradle sync** * Wait for Gradle to sync all dependencies * Resolve any dependency conflicts if prompted 4. **Run the app** * Connect your Android device or start an emulator * Click "Run" or press `Shift + F10` * Select your target device 5. **Generate slogans** * Enter a product name or description in the input field * Tap the "Generate Slogan" button * Watch as the AI creates creative marketing copy on-device * Generate multiple variations by tapping again ## Usage examples Try generating slogans for different products: **Example 1: Technology Product** ``` Input: "Wireless noise-cancelling headphones" Output: "Silence the world. Amplify your music." ``` **Example 2: Food Product** ``` Input: "Organic cold-pressed juice" Output: "Nature's energy, bottled fresh daily." ``` **Example 3: Service Business** ``` Input: "On-demand dog walking service" Output: "Happy paws, on your schedule." ``` **Example 4: Software Product** ``` Input: "AI-powered photo editing app" Output: "Your photos, brilliantly reimagined." ``` Each generation produces unique slogans tailored to your product description, perfect for brainstorming marketing campaigns. ## Understanding the architecture ### Traditional Android Views Approach Unlike examples that use Jetpack Compose, SloganApp demonstrates integration with the traditional Android View system: **XML Layout Structure:** ```xml theme={"theme":{"light":"github-light","dark":"github-dark"}}