Skip to main content
Use this guide when you want to validate LFMs on an actual product task. The goal is to compare quality, latency, cost, and integration fit against the model you would otherwise deploy.

​
Integration

LFMs serve through OpenAI-compatible endpoints on vLLM and SGLang, load in Transformers, and run locally through llama.cpp, Ollama, MLX, and LM Studio. Read these three pages before judging quality:
  • Chat template: LFMs use a ChatML-style format. Always use apply_chat_template or the server’s chat endpoint instead of hand-writing prompts.
  • Text generation and prompting: start from LFM sampling defaults instead of carrying over another model’s generation parameters.
  • Tool use: LFMs emit a Pythonic tool-call format. If your harness assumes OpenAI-style JSON tool calls, read the migration guide first.

​
What to evaluate

  • Task quality vs. your current model. Compare against the model you would actually deploy, including its cost and latency class.
  • Latency and throughput under realistic workloads. Measure your prompt lengths, output lengths, and concurrency.
  • Cost-performance. Compare tokens per second per dollar at your traffic profile.
  • Structured output and tool calling. Evaluate with the native format and constrained decoding where possible.
A fair evaluation of a small model often includes a light fine-tune. Out of the box, a 1.2B model will trail a larger generalist on broad tasks. After task-specific fine-tuning, it can match or beat that larger model on your task at lower cost and latency.

​
Evaluation checklist

  • Production-like prompts and inputs selected
  • Current production model or planned baseline selected
  • Sampling parameters reset for LFM defaults
  • Chat template generated by the tokenizer or serving runtime
  • Tool parser updated if tool calling is in scope
  • Latency measured at production p50 and p95 prompt lengths
  • Held-out examples frozen before any fine-tuning