Vision Capabilities - Liquid AI Docs

LFM vision models’ capabilities enable the model to analyze and understand images and, with LFM3-VL, videos. These models support common vision-language tasks such as describing images, answering questions about visual content, comparing multiple images, reading text, and localizing objects. They also support tool calling, including examples where an image helps determine the tool arguments. LFM3-VL is the latest generation: it reasons step by step before answering, takes video as input, and points at objects natively. The examples below show how to send images and videos to LFM3-VL models.

Using LFM2.5-VL? See LFM2.5-VL for its examples, including document layout annotations.

Install PyTorch, Transformers, and the image and video packages used by the examples. torchvision and Pillow are required by the LFM3-VL image processor, and torchcodec decodes videos. torchcodec needs FFmpeg.

# Install PyTorch and acceleration libraries
%pip install -q torch torchvision torchcodec accelerate

# Install image helper libraries
%pip install -q pillow

# Install Transformers
%pip install -q "transformers>=5.21.0"

Load the model with AutoProcessor and AutoModelForImageTextToText. LFM3-VL ships its model code in the checkpoint, so pass trust_remote_code=True.

import torch
from transformers import AutoProcessor, AutoModelForImageTextToText

MODEL_ID = "LiquidAI/LFM3-VL-2B"

processor = AutoProcessor.from_pretrained(MODEL_ID, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    MODEL_ID,
    device_map="auto",
    dtype="bfloat16",
    trust_remote_code=True,
)

LFM3-VL thinks in a <think> block before it answers, and reasoning can be long. The helper below generates with the recommended sampling parameters (temperature=0.1, top_k=50, repetition_penalty=1.1), then splits the reasoning from the answer.

def generate(messages, max_new_tokens=8192, **template_kwargs):
    inputs = processor.apply_chat_template(
        messages,
        add_generation_prompt=True,
        tokenize=True,
        return_dict=True,
        return_tensors="pt",
        **template_kwargs,
    ).to(model.device)

    with torch.inference_mode():
        outputs = model.generate(
            **inputs,
            do_sample=True,
            temperature=0.1,
            top_k=50,
            repetition_penalty=1.1,
            max_new_tokens=max_new_tokens,
        )

    text = processor.decode(outputs[0, inputs["input_ids"].shape[1]:], skip_special_tokens=False)
    text = text.replace("<|im_end|>", "").strip()
    reasoning, separator, answer = text.partition("</think>")
    if not separator:
        return "", text
    return reasoning.replace("<think>", "").strip(), answer.strip()

Single-image prompt

Provide one image and a text question in the same user message. This pattern is useful for captioning, visual question answering, and scene understanding.

from transformers.image_utils import load_image

img_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/coco_sample.png"
input_image = load_image(img_url)

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": input_image},
            {"type": "text", "text": "Describe this image in two concise sentences."},
        ],
    }
]

reasoning, answer = generate(messages)
print("Reasoning:\n", reasoning, "\n")
print("Answer:\n", answer)

Output Input image

Reasoning:
 The user wants a concise description of the image, limited to two sentences. I need to observe the key elements: two tabby cats sleeping on a pink/magenta surface (likely a couch or blanket), and two remote controls nearby. Let me look carefully.

Left cat: stretched out, head near a white remote control, eyes closed, lying on its side/back with paws extended. Right cat: larger, also tabby, curled/stretched with head down, near another remote control. The background is a bright pink fabric/couch.

I should produce two concise sentences capturing these details without over-explaining. Avoid listing every detail; keep it tight.

Draft: "Two tabby cats are asleep on a bright pink couch or blanket, each stretched out in a relaxed pose. A white remote control lies near the smaller cat's head, while a second remote rests between the two cats."

That's two sentences, accurate and concise. Good.

Answer:
 Two tabby cats are curled up asleep on a vivid pink couch or blanket, each stretched out in a relaxed, sprawling pose. A white remote control lies beside the smaller cat's head, with a second remote resting between the two cats.

Answer without reasoning

For simple questions, set enable_thinking=False to skip the reasoning and get a direct answer with lower latency.

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": input_image},
            {"type": "text", "text": "How many cats are in this image? Answer with a number."},
        ],
    }
]

_, answer = generate(messages, enable_thinking=False)
print(answer)

Output

Multi-image prompt

You can include multiple images in a single prompt. Label each image in the prompt, such as Media-1 and Media-2, to make cross-image references clearer.

img_urls = [
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG",
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/coco_sample.png",
]
input_images = [load_image(img_url) for img_url in img_urls]

messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "Media-1\n"},
            {"type": "image", "image": input_images[0]},
            {"type": "text", "text": "\nMedia-2\n"},
            {"type": "image", "image": input_images[1]},
            {"type": "text", "text": "\nCaption Media-1 and Media-2 separately. Keep each caption to one sentence."},
        ],
    }
]

_, answer = generate(messages)
print(answer)

OCR

LFM3-VL can read text in document images and screenshots.

img_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/enterprise/audit-logs.png"
document_image = load_image(img_url)

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": document_image},
            {
                "type": "text",
                "text": "Read this audit log screenshot. Identify the page heading, table columns, visible rows, users, actions, timestamps, and other structured fields. Transcribe the visible text in reading order.",
            },
        ],
    }
]

_, answer = generate(messages)
print(answer)

Output (truncated) Input image

**Page Heading:** Audit log

**Table / Section Header:** Events log  *(with a “Download as JSON” button on the right)*

...

**Visible Rows (in reading order):**

1. **clboetticher** — `org.update_join_settings`
   - *Action:* enabled join requests with manual approval and set default role to write
   - *Location:* United States, Austin · *ID:* 136.62.181.0 · *On Oct 15*

2. **clboetticher** — `repo.delete`
   - *Action:* deleted a model: enterprise-explorers/test_model
   - *Location:* United States, Austin · *ID:* 136.62.181.0 · *On Oct 10*

3. **clboetticher** — `repo.create`
   - *Action:* created a public model: enterprise-explorers/test_model
   - *Location:* United States, Austin · *ID:* 136.62.181.0 · *On Oct 10*

4. **System** — `billing.renew_subscription`
   - *Action:* System Automatic subscription renewal for hf/enterprise-free
   - *Location:* Unknown Location · *On Oct 7*

5. **derek-thomas** — `resource_group.add_users`
   - *Action:* added users to resource group Read access members: derek-thomas (read)
   - *Location:* United Arab Emirates, Abu Dhabi · *ID:* 217.164.173.0 · *On Sep 17*

6. **erikrigner** — `resource_group.change_role`
   - *Action:* changed role for user erikrigner in resource group Read access members from write to admin
   - *Location:* Sweden, Norrköping · *ID:* 2a02:1406:59d60:1dfd:0:0 · *On Sep 16*

...

Pointing

Pointing asks the model to locate visible objects with a single point each. LFM3-VL points natively as {"point_2d": [x, y]}, with x and y as integers normalized to [0, 1000]. This example asks for a point on each cat and each remote control, then draws the points.

import json
import re
from IPython.display import display
from PIL import ImageDraw

def parse_json(answer):
    match = re.search(r"\[.*\]", answer, flags=re.DOTALL)
    return json.loads(match.group(0)) if match else []

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": input_image},
            {
                "type": "text",
                "text": 'Point to each cat and each remote control. Answer in JSON as a list of points: [{"point_2d": [x, y], "label": "<short label>"}, ...], with x and y as integers normalized to 0-1000.',
            },
        ],
    }
]

_, answer = generate(messages)
print(answer)

points = parse_json(answer)
annotated = input_image.copy()
draw = ImageDraw.Draw(annotated)
width, height = annotated.size
for point in points:
    x, y = point["point_2d"]
    px, py = x / 1000 * width, y / 1000 * height
    draw.ellipse((px - 8, py - 8, px + 8, py + 8), fill="red", outline="white", width=2)
    draw.text((px + 12, py - 8), point.get("label", ""), fill="red")
display(annotated)

Output

[{"point_2d": [260, 500], "label": "cat"}, {"point_2d": [720, 400], "label": "cat"}, {"point_2d": [168, 195], "label": "remote control"}, {"point_2d": [550, 270], "label": "remote control"}]

Points drawn on the cats and remote controls

Object detection and grounding

Grounding asks the model to localize visible objects with bounding boxes. This example uses the same image, asks for boxes in [xmin, ymin, xmax, ymax] format normalized to [0, 1000], then draws the returned boxes.

Coordinate format: Convert normalized coordinates back to pixels with x / 1000 * width and y / 1000 * height. Points and boxes are approximate, so visually inspect the result before using coordinates in downstream workflows.

system_prompt = """When asked for bounding boxes for objects, return a valid JSON array.
Each array item must be an object with:
- bbox_2d: [xmin, ymin, xmax, ymax] normalized integer coordinates in [0, 1000]
- label: a concise label you choose for the predicted object or region

Return one item per visible matching object or region. Return [] if none are visible."""

messages = [
    {"role": "system", "content": system_prompt},
    {
        "role": "user",
        "content": [
            {"type": "image", "image": input_image},
            {"type": "text", "text": "Provide bounding boxes for the two cats and the two remote controls."},
        ],
    },
]

_, answer = generate(messages)
print(answer)

objects = parse_json(answer)
annotated = input_image.copy()
draw = ImageDraw.Draw(annotated)
for obj in objects:
    xmin, ymin, xmax, ymax = obj["bbox_2d"]
    box = (xmin / 1000 * width, ymin / 1000 * height, xmax / 1000 * width, ymax / 1000 * height)
    draw.rectangle(box, outline="red", width=3)
    draw.text((box[0] + 4, box[1] + 4), obj.get("label", ""), fill="red")
display(annotated)

Output

[{"bbox_2d": [15, 113, 497, 987], "label": "cat"}, {"bbox_2d": [536, 46, 999, 776], "label": "cat"}, {"bbox_2d": [62, 145, 276, 245], "label": "remote control"}, {"bbox_2d": [519, 156, 585, 393], "label": "remote control"}]

Bounding boxes drawn around the cats and remote controls

Video understanding

LFM3-VL takes video as input. Videos are processed at their native resolution: they are sampled at 2 frames per second, consecutive frames are paired, and each frame is prefixed with its timestamp. Pass a local video path with {"type": "video", "video": ...}. This example uses a short clip from the Hugging Face test videos.

from huggingface_hub import hf_hub_download

video_path = hf_hub_download("raushan-testing-hf/videos-test", "sample_demo_1.mp4", repo_type="dataset")

messages = [
    {
        "role": "user",
        "content": [
            {"type": "video", "video": video_path},
            {"type": "text", "text": "Describe what happens in this video."},
        ],
    }
]

_, answer = generate(messages)
print(answer)

Output

In this short home-video clip, a young blonde toddler is sitting on a bed covered with a patterned quilt, apparently "reading" a small paperback book. The baby is dressed in a light blue/mint sleeveless top and pink pants, and is wearing a pair of large dark-framed eyeglasses that make her look like a little reader. Behind her you can see a wooden crib or chair and some folded clothes and fabric on the bed.

At the start, the child holds the open book in her lap and gazes down at the pages. Around the middle of the clip she becomes animated—she lifts her hands and claps/waves them excitedly, as if reacting to something or getting absorbed in her activity. After a moment she brings her hands back down to the book and begins turning the pages one by one, looking attentively at the text as though reading. She continues to flip through the pages and study them until the end of the clip. Overall, the video shows a baby wearing glasses sitting on a bed and pretending to read, pausing briefly to clap her hands before going back to turning the pages of the book.

To also use a video’s soundtrack, use LFM3-Omni-2B. See Omni Capabilities.

Tool calling

In this example, the model sees an image and can route the request to the most relevant tool. Pass the tool schemas to apply_chat_template() with tools=.... LFM3-VL writes each call between <tool_call> and </tool_call>, with the function name in a <function=...> tag and each argument in a <parameter=...> tag.

tools = [
    {
        "type": "function",
        "function": {
            "name": "search_pet_care",
            "description": "Search for care guidance for an animal visible in an image.",
            "parameters": {
                "type": "object",
                "properties": {
                    "animal": {"type": "string", "description": "The animal visible in the image"},
                    "topic": {"type": "string", "description": "The care topic to search for"},
                },
                "required": ["animal", "topic"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "search_replacement_remote",
            "description": "Search for replacement remote controls or remote-control accessories visible in an image.",
            "parameters": {
                "type": "object",
                "properties": {
                    "item": {"type": "string", "description": "The remote-control item to search for"},
                    "quantity": {"type": "integer", "description": "How many matching items are visible"},
                },
                "required": ["item"],
            },
        },
    },
]

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": input_image},
            {"type": "text", "text": "Find a care guide for the animals in this image. Choose the best tool for this request."},
        ],
    }
]

_, answer = generate(messages, tools=tools)
print(answer)

Output

<tool_call>
<function=search_pet_care>
<parameter=animal>
cat
</parameter>
<parameter=topic>
care
</parameter>
</function>
</tool_call>

Execute the call in your application, add the parsed call to the conversation as a tool_calls entry, and return the result as a tool message. See Tool Use for the full workflow.

LFM2.5-VL

LFM2.5-VL models answer directly without a reasoning trace and use the LFM2.5 tool call format. LFM2.5-VL-3B also returns document layout annotations. Install PyTorch, Transformers, and the image-processing packages used by the examples. torchvision and Pillow are required by the LFM2.5-VL image processor, and opencv-python is used to draw grounding and layout boxes.

# Install PyTorch and acceleration libraries
%pip install -q torch torchvision accelerate

# Install image helper libraries
%pip install -q pillow opencv-python

# Install Transformers
%pip install -q "transformers>=5.10.1"

Load the model with AutoProcessor and AutoModelForImageTextToText, matching the standard LFM2.5-VL documentation examples. Start with LiquidAI/LFM2.5-VL-450M or LiquidAI/LFM2.5-VL-1.6B for fast iteration. Use LFM2.5-VL-3B for the strongest grounding, layout parsing, and tool-calling examples.

from transformers import AutoProcessor, AutoModelForImageTextToText

MODEL_ID = "LiquidAI/LFM2.5-VL-3B"  # "LiquidAI/LFM2.5-VL-450M", "LiquidAI/LFM2.5-VL-1.6B", "LiquidAI/LFM2.5-VL-3B"

processor = AutoProcessor.from_pretrained(MODEL_ID)
model = AutoModelForImageTextToText.from_pretrained(
    MODEL_ID,
    device_map="auto",
    dtype="bfloat16",
)

LFM2.5-VL single-image prompt

Provide one image and a text question in the same user message. This pattern is useful for captioning, visual question answering, and scene understanding.

import torch
from IPython.display import display
from transformers.image_utils import load_image

img_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/coco_sample.png"
input_image = load_image(img_url)
display(input_image)

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": input_image},
            {"type": "text", "text": "Describe this image in two concise sentences."},
        ],
    }
]

inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        do_sample=True,
        temperature=0.2,
        top_k=50,
        repetition_penalty=1.0,
        max_new_tokens=256,
    )

output = processor.batch_decode(outputs[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0]
print(output)

Output Cell 7 output

Two cats are sleeping on a pink couch with two remote controls.

LFM2.5-VL multi-image prompt

Model support: This capability is best supported by LFM2.5-VL-3B.

You can include multiple images in a single prompt. Label each image in the prompt, such as Media-1 and Media-2, to make cross-image references clearer.

import torch
from IPython.display import display
from transformers.image_utils import load_image

img_urls = [
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG",
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/coco_sample.png",
]
input_images = [load_image(img_url) for img_url in img_urls]
for input_image in input_images:
    display(input_image)

messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "Media-1\n"},
            {"type": "image", "image": input_images[0]},
            {"type": "text", "text": "\nMedia-2\n"},
            {"type": "image", "image": input_images[1]},
            {"type": "text", "text": "\nCaption Media-1 and Media-2 separately. Keep each caption to one sentence."},
        ],
    }
]

inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        do_sample=True,
        temperature=0.2,
        top_k=50,
        repetition_penalty=1.0,
        max_new_tokens=256,
    )

output = processor.batch_decode(outputs[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0]
print(output)

Output

Media-1: A person is holding a collection of colorful M&M's with various designs on them.

Media-2: Two cats are peacefully sleeping on a pink couch, each with a remote control nearby.

LFM2.5-VL OCR

Model support: This capability is best supported by LFM2.5-VL-3B.

LFM2.5-VL can read text in document images and, with LFM2.5-VL-3B, return structured layout annotations, which can be used to visualize the parsed layout regions.

Image quality: OCR and layout parsing work best when text is legible and the image is not blurry, rotated, or heavily compressed. For dense documents, crop to the relevant page or region when possible.

import torch
from IPython.display import display
from transformers.image_utils import load_image

img_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/enterprise/audit-logs.png"
input_image = load_image(img_url)
display(input_image)

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": input_image},
            {
                "type": "text",
                "text": "Read this audit log screenshot. Identify the page heading, table columns, visible rows, users, actions, timestamps, and other structured fields. Transcribe the visible text in reading order.",
            },
        ],
    }
]

inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        do_sample=True,
        temperature=0.1,
        top_k=50,
        repetition_penalty=1.05,
        max_new_tokens=512,
    )

generated_ids = outputs[:, inputs["input_ids"].shape[1] :]
output = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(output)

Output Cell 11 output

Audit log

| Events log |
| --- |
| clboetticher org.update_join_settings |
| clboetticher enabled join requests with manual approval and set default role to write |
| United States, Austin · 136.62.181.0 · On Oct 15 |
| clboetticher repo.delete |
| clboetticher deleted a model: enterprise-explorers/test_model |
| United States, Austin · 136.62.181.0 · On Oct 10 |
| clboetticher repo.create |
| clboetticher created a public model: enterprise-explorers/test_model |
| United States, Austin · 136.62.181.0 · On Oct 10 |
| System billing.renew_subscription |
| System Automatic subscription renewal for hf/enterprise-free |
| Unknown Location · On Oct 7 |
| derek-thomas resource_group.add_users |
| derek-thomas added users to resource group Read access members: derek-thomas (read) |
| United Arab Emirates, Abu Dhabi · 217.164.173.0 · On Sep 17 |
| erikrigner resource_group.change_role |
| erikrigner changed role for user erikrigner in resource group Read access members from write to admin |
| Sweden, Norrköping · 2a02:1406:59:5d60:1dfd:0:0 · On Sep 16 |

Document layout annotations

For document understanding tasks, LFM2.5-VL-3B can return OCR with layout annotations. Each region contains a label, normalized bounding box, and content:

image_index=<n> <label> [xmin, ymin, xmax, ymax]
<content>

Coordinates are normalized integers in [0, 1000], matching the grounding format. If you want to visualize these boxes, scale the coordinates back to pixels with the same x / 1000 * width and y / 1000 * height pattern used in the grounding example.

import re
import torch
import cv2
import numpy as np
from IPython.display import display
from PIL import Image
from transformers.image_utils import load_image

img_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/enterprise/audit-logs.png"
input_image = load_image(img_url)
display(input_image)

layout_prompt = """Parse this document into its layout regions. The pages are provided as images in reading order. For every region, in reading order across all pages, output a header line immediately followed by the region's content:

image_index=<n> <label> [xmin, ymin, xmax, ymax]
<content>

where:
- image_index is the zero-based index of the page image the region appears on (0 for the first image, 1 for the second, and so on)
- <label> is one of these layout labels: text, title, list, table, table_caption, table_footnote, image, image_block, image_caption, image_footnote, chart, equation, formula_number, code, code_caption, algorithm, aside_text, ref_text, phonetic, page_header, page_footer, page_number, page_footnote
- [xmin, ymin, xmax, ymax] are normalized integer coordinates in [0, 1000]
- <content> is the region's content: plain text for text regions, LaTeX for equations, OTSL for tables, and a short description for images and charts

Separate each region block with one blank line. Return only the parsed regions.
"""

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": input_image},
            {"type": "text", "text": layout_prompt},
        ],
    }
]

inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        do_sample=True,
        temperature=0.2,
        top_k=50,
        repetition_penalty=1.0,
        max_new_tokens=1024,
    )

output = processor.batch_decode(
    outputs[:, inputs["input_ids"].shape[1]:],
    skip_special_tokens=True,
)[0]
print(output)

pattern = re.compile(
    r"image_index=0\s+([a-z_]+)\s+\[(\d+),\s*(\d+),\s*(\d+),\s*(\d+)\]"
)

image = np.array(input_image).copy()
height, width = image.shape[:2]

for label, xmin, ymin, xmax, ymax in pattern.findall(output):
    xmin, ymin, xmax, ymax = map(int, [xmin, ymin, xmax, ymax])
    pt1 = (round(xmin / 1000 * width), round(ymin / 1000 * height))
    pt2 = (round(xmax / 1000 * width), round(ymax / 1000 * height))
    cv2.rectangle(image, pt1, pt2, (255, 0, 0), 2)
    cv2.putText(image, label, (pt1[0], max(20, pt1[1] - 6)), cv2.FONT_HERSHEY_SIMPLEX, 0.6, (255, 0, 0), 2)

display(Image.fromarray(image))

Output

image_index=0 label [76, 68, 144, 94]
Acme Inc.

image_index=0 text [78, 104, 104, 121]
acme

image_index=0 text [28, 176, 65, 197]
Profile

image_index=0 text [28, 230, 74, 252]
Account

image_index=0 text [28, 282, 80, 304]
Members

image_index=0 text [28, 334, 64, 355]
Billing

... (shortened for brevity)

LFM2.5-VL object detection and grounding

Model support: This capability is best supported by LFM2.5-VL-3B.

Grounding asks the model to localize visible objects and return normalized coordinates. This example uses a COCO sample image with cats and remote controls, then draws the returned boxes with OpenCV.

Coordinate format: The grounding examples use normalized [0, 1000] coordinates. Convert them back to pixels with x / 1000 * width and y / 1000 * height. Boxes are approximate, so visually inspect the result before using coordinates in downstream workflows.

import json
import cv2
import numpy as np
import torch
from IPython.display import display
from PIL import Image
from transformers.image_utils import load_image

img_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/coco_sample.png"
input_image = load_image(img_url)

system_prompt = """When asked for bounding boxes for objects, return a valid JSON array.
Each array item must be an object with:
- image_id: the 0-based index of the image
- bbox_2d: [xmin, ymin, xmax, ymax] normalized integer coordinates in [0, 1000]
- label: a concise label you choose for the predicted object or region

Return one item per visible matching object or region. Return [] if none are visible."""

messages = [
    {"role": "system", "content": system_prompt},
    {
        "role": "user",
        "content": [
            {"type": "image", "image": input_image},
            {"type": "text", "text": "Provide bounding boxes for the two cats and the two remote controls."},
        ],
    },
]

inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        do_sample=True,
        temperature=0.2,
        top_k=50,
        repetition_penalty=1.0,
        max_new_tokens=256,
    )

output = processor.batch_decode(
    outputs[:, inputs["input_ids"].shape[1]:],
    skip_special_tokens=True,
)[0]
print(output)

objects = json.loads(output)
print(objects)

image = np.array(input_image).copy()
height, width = image.shape[:2]

for obj in objects:
    xmin, ymin, xmax, ymax = obj["bbox_2d"]
    pt1 = (round(xmin / 1000 * width), round(ymin / 1000 * height))
    pt2 = (round(xmax / 1000 * width), round(ymax / 1000 * height))
    cv2.rectangle(image, pt1, pt2, color=(255, 0, 0), thickness=3)
    cv2.putText(image, obj["label"], (pt1[0], max(20, pt1[1] - 8)), cv2.FONT_HERSHEY_SIMPLEX, 0.8, (255, 0, 0), 2)

display(Image.fromarray(image))

Output

[{"image_id": 0, "bbox_2d": [540, 49, 1000, 774], "label": "cat on the right"}, {"image_id": 0, "bbox_2d": [10, 112, 497, 986], "label": "cat on the left"}, {"image_id": 0, "bbox_2d": [523, 158, 581, 398], "label": "remote on the right"}, {"image_id": 0, "bbox_2d": [63, 150, 276, 244], "label": "remote on the left"}]
[{'image_id': 0, 'bbox_2d': [540, 49, 1000, 774], 'label': 'cat on the right'}, {'image_id': 0, 'bbox_2d': [10, 112, 497, 986], 'label': 'cat on the left'}, {'image_id': 0, 'bbox_2d': [523, 158, 581, 398], 'label': 'remote on the right'}, {'image_id': 0, 'bbox_2d': [63, 150, 276, 244], 'label': 'remote on the left'}]

LFM2.5-VL tool calling

Model support: This capability is best supported by LFM2.5-VL-3B.

In this example, the model sees an image and can route the request to the most relevant tool. The tool schemas are generic enough to adapt to real catalog, search, or workflow APIs.

import torch
from IPython.display import display
from transformers.image_utils import load_image

img_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/coco_sample.png"
input_image = load_image(img_url)
display(input_image)

tools = [
    {
        "name": "search_pet_care",
        "description": "Search for care guidance for an animal visible in an image.",
        "parameters": {
            "type": "object",
            "properties": {
                "animal": {"type": "string", "description": "The animal visible in the image"},
                "topic": {"type": "string", "description": "The care topic to search for"},
            },
            "required": ["animal", "topic"],
        },
    },
    {
        "name": "search_replacement_remote",
        "description": "Search for replacement remote controls or remote-control accessories visible in an image.",
        "parameters": {
            "type": "object",
            "properties": {
                "item": {"type": "string", "description": "The remote-control item to search for"},
                "quantity": {"type": "integer", "description": "How many matching items are visible"},
            },
            "required": ["item"],
        },
    },
]

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": input_image},
            {"type": "text", "text": "Find a care guide for the animals in this image. Choose the best tool for this request."},
        ],
    }
]

inputs = processor.apply_chat_template(
    messages,
    tools=tools,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        do_sample=True,
        temperature=0.2,
        top_k=50,
        repetition_penalty=1.0,
        max_new_tokens=256,
    )

output = processor.tokenizer.decode(
    outputs[0, inputs["input_ids"].shape[1]:],
    skip_special_tokens=False,
)
print(output)

Output Cell 17 output

<|tool_call_start|>[search_pet_care(animal="cat", topic="sleeping")]<|tool_call_end|><|im_end|>

FAQ

Which LFM vision model should I use? Use LFM3-VL-2B for most use cases, including video understanding, harder visual questions that benefit from reasoning, pointing, and grounding. Among LFM2.5-VL models, use LFM2.5-VL-450M or LFM2.5-VL-1.6B for fast image captioning and visual question answering, and LFM2.5-VL-3B for document layout parsing. Can I turn off reasoning? Yes, for LFM3-VL. Pass enable_thinking=False to apply_chat_template() for direct answers with lower latency. LFM2.5-VL models always answer directly. What image and video formats are supported? LFM vision models work with common image formats such as JPEG and PNG. In the examples above, images are loaded from Hugging Face URLs with transformers.image_utils.load_image. LFM3-VL also takes video files that torchcodec can decode, such as MP4. Can the models point at or localize objects? Yes. Prompt LFM3-VL to return points as {"point_2d": [x, y]} or bounding boxes as bbox_2d: [xmin, ymin, xmax, ymax], with coordinates normalized to [0, 1000]. LFM2.5-VL returns bounding boxes in the same format. Boxes and points are approximate and can vary with prompt wording, image resolution, and generation settings. For workflows that require precise localization, inspect the returned coordinates and validate them against the original image. Can the models parse documents? Yes. LFM3-VL reads visible text in documents and screenshots. LFM2.5-VL-3B can also return document layout annotations, including region labels, bounding boxes, and extracted content. Can the models call tools based on an image? Yes. You can provide tool schemas through the chat template and ask the model to choose the relevant tool based on the visual content. LFM3-VL uses the LFM3 <tool_call> format, and LFM2.5-VL uses the LFM2.5 format. See Tool Use. Can the models generate or edit images? No. LFM vision models are image and video understanding models. They can analyze images and videos, answer questions, read text, localize objects, parse layouts, and route tool calls, but they do not generate or edit images.

​Single-image prompt

​Answer without reasoning

​Multi-image prompt

​OCR

​Pointing

​Object detection and grounding

​Video understanding

​Tool calling

​LFM2.5-VL

​LFM2.5-VL single-image prompt

​LFM2.5-VL multi-image prompt

​LFM2.5-VL OCR

​Document layout annotations

​LFM2.5-VL object detection and grounding

​LFM2.5-VL tool calling

​FAQ

Single-image prompt

Answer without reasoning

Multi-image prompt

OCR

Pointing

Object detection and grounding

Video understanding

Tool calling

LFM2.5-VL

LFM2.5-VL single-image prompt

LFM2.5-VL multi-image prompt

LFM2.5-VL OCR

Document layout annotations

LFM2.5-VL object detection and grounding

LFM2.5-VL tool calling

FAQ