Small Language Models (SLMs): What They Are, Which to Pick, and When to Use Them Instead of an LLM

Summary

Small language models (SLMs) share their foundations with large language models (LLMs) but are kept small enough to run on a laptop, a phone or a single GPU. The reason everyone is talking about them in 2026 is simple: a lot of real work does not need a model with hundreds of billions of parameters. Tagging, extraction, routing, short summaries and the repetitive tool calls inside an agent can be done by a well-chosen small model more cheaply, more quickly and without the data leaving your hands.

This guide first pins down what "small" means, then lists the notable open-weight families as of October 2026, with sizes and licenses. After that: an SLM vs LLM decision table, the techniques that make small models good, on-device runtimes, a routing pattern that combines an LLM and an SLM, and how to evaluate for your own task.

Note: Model names, sizes and licenses were checked against Hugging Face model cards and vendor pages as of October 24, 2026. Speed and quality figures are the vendors' own measurements, and we say whose they are. This space moves fast; read the current model card before you decide.

How small is "small"?

There is no agreed-upon parameter cutoff for an SLM. Three perspectives are in use, and each is reasonable on its own terms:

• Functional definition. NVIDIA Research's "Small Language Models are the Future of Agentic AI" (Belcak et al., first version June 2025, latest version September 2026) defines an SLM as a language model that fits on a common consumer device and answers one user's requests with practical latency. The definition is deliberately time-dependent: what is "large" today may be "small" tomorrow as hardware improves.
• Rough parameter range. The same paper says that "as of 2025, we would be comfortable with considering most models below 10bn parameters" to be SLMs. In industry usage the ~0.1B–10B range is what people usually mean. 12–14B models (e.g. Ministral 3 14B, Gemma 4 12B) sit in a grey zone.
• Deployment target. Some vendors talk about target devices rather than size: Google positions Gemma 4 E2B and E4B for "mobile and edge devices" and 12B and up for "consumer GPUs and workstations".

Parameter count on its own can mislead. Two examples:

• "Effective" parameters: Gemma 4 E2B's model card reports 2.3B effective and 5.1B total parameters including embeddings. The "E" comes from Per-Layer Embeddings: those tables are large but used only for fast lookups.
• Mixture of Experts: Liquid AI's LFM2.5-8B-A1B has ~8B parameters in total, but "A1B" in the name says roughly 1B are active per token. In MoE models speed tracks active parameters while memory tracks total parameters.

Rule of thumb: instead of "is this model small?", ask "is this model good enough on my hardware, within my latency budget and on my task?"

Why is this trending in 2026?

1. Cost. Paying an API per request becomes the biggest line item as traffic grows. The NVIDIA paper states that serving a 7B SLM is 10–30× cheaper in latency, energy and FLOPs than a 70–175B LLM (an estimate the authors compile from the literature, not your measurement). The exact ratio depends on your hardware and traffic, but the direction is clear.

2. Latency. A small model does less compute and can run close to the user (on the device or in the same data center). Removing the network round trip makes autocomplete, voice interfaces and real-time classification practical.

3. On-device. Phone and laptop hardware can now carry quantized 1–4B models. Apple has said Apple Intelligence is built around a ~3B-parameter on-device model, compressed with 2-bit quantization-aware training, and that its Foundation Models framework, introduced in 2025, gives app developers direct access to it. Google writes that Gemma 4 E2B/E4B are designed "for efficient local execution on laptops and mobile devices".

4. Privacy and data sovereignty. If the model runs on your device or server, prompts and documents don't have to leave. In health, legal, finance and internal company data, that alone can decide the matter. (See AI data privacy.)

5. Agents and routing. An AI agent makes dozens of model calls per task, and most of them are narrow and repetitive: "turn this output into JSON", "which tool should I call", "is this document relevant". That is the NVIDIA paper's core thesis: a large share of calls inside agents can be handed to SLMs, with heterogeneous systems (models of different sizes working together) where general conversational ability is needed.

Notable SLM families as of October 2026

The table contains only what we could verify on Hugging Face model cards. The "sizes" column uses the card's figures; numbers in parentheses are totals computed by Hugging Face from the safetensors files.

A few practical lessons from the table:

• The license narrows your choice. Apache 2.0 and MIT are the lowest-friction options for commercial use. The LFM Open License v1.0 excludes commercial use by companies with $10M or more in annual revenue. Llama 3.2 requires access approval on Hugging Face and comes with its own license terms.
• "Small" can be multimodal now. Gemma 4 E2B/E4B take audio and images; Qwen3.5 and Ministral 3 take images. A model that transcribes speech or reads a photo on a phone now lives at 2–4B scale.
• Long context ≠ cheap context. 128K–262K context is possible on paper, but KV cache memory grows with context length. Practical context on a phone will be much shorter than the card says.
• Benchmarks are the vendors'. Nearly every score table on a model card is the vendor's own measurement. Liquid AI reports 239 tokens/s on an AMD CPU, 82 tokens/s on a mobile NPU and under 1 GB of memory for LFM2.5-1.2B; Mistral says the 3B fits in 8 GB of VRAM. Treat these as starting points, not decision data.

SLM or LLM? A decision table

If both columns light up (as they do in most real systems), the answer is usually both: a routing layer that sends routine requests to the SLM and hard or uncertain ones to the LLM. An example follows below.

Techniques that make small models good

Distillation

Distillation trains a small "student" model on the outputs of a large "teacher". The idea goes back to Hinton, Vinyals and Dean's 2015 paper "Distilling the Knowledge in a Neural Network": the student learns not just the right answer but the teacher's probability distribution. Today it shows up in two forms:

• Logit/probability distillation: the student imitates the teacher's token probabilities. This needs access to the teacher's weights or output probabilities.
• Distillation via synthetic data: a large model generates examples and answers for a task, and the small model is fine-tuned on that data. Microsoft's Phi-4-mini card says the model is built on "synthetic data and filtered publicly available websites"; IBM includes internally generated synthetic data in Granite 4.2's training mix.

Caution: training on a commercial API's outputs can conflict with that provider's terms of use. Check the teacher's license and terms.

Quantization

Quantization stores weights in 8 or 4 bits instead of 16. Memory shrinks roughly in proportion to the bit width, and speed usually improves. A rough, weights-only estimate:

The table covers weights only. On top come the KV cache (which grows with context length), runtime overhead and, for multimodal models, the vision/audio encoders. Run the numbers for your own hardware with the GPU / VRAM calculator.

A real-world sanity check: the 4-bit (q4_K_M) files in the Ollama library match this table for text-only models: Phi-4-mini (3.8B) 2.5 GB, Granite 4.2 3B 2.2 GB, Granite 4.2 8B 5.3 GB. Multimodal models are noticeably larger: Gemma 4 E2B 4.6 GB, E4B 6.6 GB, Qwen3.5-9B 6.6 GB, Ministral 3 3B 3.0 GB. The difference comes from embedding tables and vision/audio encoders; the "2B" in Gemma 4 E2B does not mean 2B worth of memory on a phone.

Several vendors publish quantized builds themselves: Google ships QAT (quantization-aware training) q4_0 GGUF files for Gemma 4; Qwen ships FP8 and GPTQ-Int4 for its larger Qwen3.5 sizes; Mistral ships FP8 and GGUF; IBM ships FP8 and GGUF. Prefer the vendor's own quantized build where possible; community builds vary in quality.

Fine-tuning with LoRA

A small model's real strength is that it can be cheaply adapted to your task. LoRA (Hu et al., 2021) freezes the model's weights and injects small trainable low-rank matrices into each layer; on GPT-3 175B the paper reports 10,000× fewer trainable parameters and 3× less GPU memory. QLoRA (Dettmers et al., 2023) combines this with a 4-bit quantized base model; the paper shows fine-tuning a 65B model on a single 48 GB GPU. For a 3–4B model that means LoRA training on a single consumer-class GPU.

When to fine-tune? When prompting and a few examples can't reach the quality you need, you have a few hundred to a few thousand good examples, and the task is stable. It works best for teaching format and behavior, not for adding knowledge; for up-to-date facts RAG is the better fit.

Running on devices and at the edge

Where you run the SLM decides which tools you use.

Trying a few small models with Ollama is one line each (tags from ollama.com/library, October 2026):

ollama run gemma4:e2b
ollama run qwen3.5:4b
ollama run ministral-3:3b
ollama run granite4.2:3b
ollama run phi4-mini

For the details of running models locally (hardware choice, tool comparison, setup), see the sibling guide: Running an LLM locally. Per-tool pages live in the Local AI section.

Pattern: LLM + SLM routing and cascades

The most common production pattern is not "send everything to the small model" but try the small model first, escalate when needed. The idea has two academic foundations:

• FrugalGPT (Chen, Zaharia, Zou, 2023): proposes an LLM cascade that sends queries to a cheap model first and moves up to a more expensive one if the answer is not good enough.
• RouteLLM (Ong et al., 2024): a router trained on human preference data decides, per query, whether to use the strong or the weak model.

A simple cascade looks like this: the SLM produces a structured answer, code validates it, and if it doesn't match the schema or the model says "unsure", the request goes to the large model.

# pip install openai
import json, os
from openai import OpenAI

# Local SLM: Ollama's OpenAI-compatible endpoint
slm = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
# Cloud LLM: any OpenAI-compatible provider
llm = OpenAI(base_url=os.environ["CLOUD_BASE_URL"], api_key=os.environ["CLOUD_API_KEY"])

LABELS = {"billing", "technical", "sales", "unsure"}
PROMPT = (
    "Classify the support message. Return JSON only: "
    '{"label": "billing|technical|sales|unsure"}. If you are not sure, say "unsure".\n\nMessage: '
)

def ask(client, model, text):
    r = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": PROMPT + text}],
        temperature=0,
    )
    return r.choices[0].message.content

def classify(text):
    try:
        label = json.loads(ask(slm, "qwen3.5:4b", text))["label"]
        if label in LABELS and label != "unsure":
            return label, "slm"
    except (json.JSONDecodeError, KeyError, TypeError):
        pass  # malformed output = escalate
    label = json.loads(ask(llm, os.environ["CLOUD_MODEL"], text))["label"]
    return label, "llm"

print(classify("I was charged twice on my March invoice."))

Things to watch in this pattern:

• The escalation signal matters. Schema validation, an allowed label set and an "unsure" option are the simplest signals. Better: a separate verifier, or a decision model that returns calibrated probabilities (see System One models).
• Measure the ratio. The real metrics are "what share of requests stayed on the SLM" and "how accurate were those". In the first week, run both models in parallel and compare.
• Don't lose privacy at escalation. If you picked an SLM so data stays on the device, escalating to a cloud model breaks that promise. Design user consent, masking, or the option to disable escalation entirely, from the start.
• Split agents into subtasks. Letting a large model plan while an SLM handles format conversion, tool-argument extraction or document filtering is the practical form of the heterogeneous setup the NVIDIA paper argues for.

How to evaluate for your task

General benchmark scores are a weak signal for ranking small models. The real decision has to be measured on your task.

1. Build an evaluation set. Pick 50–200 examples from real traffic and write the correct answers by hand. Mix easy, typical and hard cases; if you serve several languages, include each one, with abbreviations and typos.

2. Set a baseline. Run the same set on the large model you use today. The goal is not "as good as the LLM" but "as good as the job requires".

3. Pick two or three candidates. Drop anything your license or hardware rules out. A typical shortlist: one 1–2B (e.g. Qwen3.5-2B, LFM2.5-1.2B), one 3–4B (e.g. Gemma 4 E4B, Ministral 3 3B, Granite 4.2 3B, Phi-4-mini) and one 8–9B (e.g. Qwen3.5-9B, Ministral 3 8B).

4. Measure in the target environment. Alongside quality, record:

• Time to first token and tokens per second, on the target hardware (on the phone if it will run on a phone)
• Peak memory at the context length you'll actually use
• Quality of the quantized build, on the same set as the original
• The rate of malformed or off-schema outputs

5. Look at error types. Average accuracy is not enough. Where does the small model fail? Can a rule or escalation catch those cases? If errors cluster in a predictable subset, the routing pattern works well.

6. If it isn't good enough, try in order: better prompt and a few examples → context via RAG → LoRA fine-tuning → the next size up. Moving up a size is usually the most expensive step, so leave it for last.

Checklist

• Does the license fit my commercial use (revenue thresholds, gated access)?
• Does the model card list my target languages, and did I test with my own examples?
• Does my memory estimate include the KV cache and multimodal encoders?
• Did I compare the quantized build with the original on the same set?
• Is the escalation (fallback) path and its rate monitored?
• Is the model version pinned (a specific version or file hash rather than a tag)?

Further reading

• Small Language Model — the short definition.
• Quantization, Distillation, LoRA — shrinking and adapting techniques.
• Mixture of Experts and KV Cache — two concepts that change the memory and speed math.
• Running an LLM locally — hardware and tool choices.
• System One models — single-pass decision models for routing decisions.
• Do you need AI, ML, or just an LLM? — sometimes the right answer is a classic classifier.

Sources

• Belcak et al. (NVIDIA Research), Small Language Models are the Future of Agentic AI, arXiv:2506.02153 — https://arxiv.org/abs/2506.02153
• Gemma 4 E2B model card — https://huggingface.co/google/gemma-4-E2B-it · Gemma release notes — https://ai.google.dev/gemma/docs/releases
• Qwen3.5-0.8B and Qwen3.5-9B model cards — https://huggingface.co/Qwen/Qwen3.5-0.8B, https://huggingface.co/Qwen/Qwen3.5-9B · Qwen3.5 GitHub README (release notes) — https://github.com/QwenLM/Qwen3.5
• Ministral 3 3B and 14B model cards — https://huggingface.co/mistralai/Ministral-3-3B-Instruct-2512, https://huggingface.co/mistralai/Ministral-3-14B-Instruct-2512 · Mistral, Introducing Mistral 3 — https://mistral.ai/news/mistral-3
• Granite 4.2 3B model card — https://huggingface.co/ibm-granite/granite-4.2-3b · IBM, Granite 4.2 LLMs: How They're Built — https://huggingface.co/blog/ibm-granite/granite-4-2
• Phi-4-mini-instruct model card — https://huggingface.co/microsoft/Phi-4-mini-instruct
• SmolLM3-3B model card — https://huggingface.co/HuggingFaceTB/SmolLM3-3B
• LFM2.5-1.2B-Instruct model card and LFM Open License v1.0 — https://huggingface.co/LiquidAI/LFM2.5-1.2B-Instruct
• Llama 3.2 1B Instruct — https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct · Llama 3.2 model card (GitHub) — https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/MODEL_CARD.md · Meta, Llama 3.2 announcement — https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/
• Apple, Apple Intelligence Foundation Language Models 2025 — https://machinelearning.apple.com/research/apple-foundation-models-2025-updates
• Hinton, Vinyals, Dean, Distilling the Knowledge in a Neural Network (2015) — https://arxiv.org/abs/1503.02531
• Hu et al., LoRA (2021) — https://arxiv.org/abs/2106.09685 · Dettmers et al., QLoRA (2023) — https://arxiv.org/abs/2305.14314
• Chen, Zaharia, Zou, FrugalGPT (2023) — https://arxiv.org/abs/2305.05176 · Ong et al., RouteLLM (2024) — https://arxiv.org/abs/2406.18665
• Runtimes: https://github.com/google-ai-edge/LiteRT-LM · https://github.com/pytorch/executorch · https://github.com/mlc-ai/mlc-llm · https://github.com/microsoft/onnxruntime-genai · https://github.com/huggingface/transformers.js
• Ollama model library and tag sizes — https://ollama.com/library (e.g. https://ollama.com/library/gemma4/tags)

Frequently asked questions

What is an SLM?

An SLM (small language model) is a language model with the same architecture as a large language model but kept small enough to run on a laptop, a phone or a single GPU. In practice the term usually refers to models between a few hundred million and roughly 10 billion parameters.

How many parameters does a small language model have?

There is no universal cutoff. NVIDIA Research's SLM paper uses a functional definition (a model that fits on a consumer device and runs with practical latency) and says it would consider most models below 10 billion parameters to be SLMs; 12–14B models are called small by some and medium by others.

Can a small language model run on a phone?

Yes. Models around 1–4B parameters, quantized to 4-bit, can run on current phones. Gemma 4 E2B/E4B and LFM2.5 are explicitly designed by their vendors for mobile and edge use, and runtimes such as Google's LiteRT-LM, PyTorch ExecuTorch and MLC LLM target this; speed and memory vary by device, so measure on yours.

Should I use an SLM or an LLM?

Try an SLM first when the task is narrow, repetitive and well defined, when data must stay on the device, or when latency is critical. For broad world knowledge, long reasoning or vague open-ended tasks a large model is safer, and many systems combine both behind a router.

What is the best small language model?

There is no single best model; it depends on your task, languages, hardware and license needs. In 2026 the Gemma 4, Qwen3.5, Ministral 3, Granite 4.2, Phi-4-mini, SmolLM3 and LFM2.5 families stand out, and comparing two or three candidates on your own 50–200 example evaluation set is the most reliable way to choose.

Are small language models free?

Most open-weight SLMs are free to download: Qwen3.5, Gemma 4, Ministral 3, Granite 4.2 and SmolLM3 are Apache 2.0 and Phi-4-mini is MIT. Llama 3.2 uses its own community license and LFM2.5 uses a license that restricts commercial use by companies with $10M or more in annual revenue, and you still pay for hardware and operations.

Are SLMs better for privacy?

They can be, because a model that runs on your own device or server means prompts and documents never have to leave it. The privacy benefit only holds if the whole pipeline stays local, including logs, telemetry and any fallback to a cloud model.