AI Atlas
Intermediate · ~2 min read #num_ctx #context-length #inference

num_ctx

Runtime context length parameter

On local inference engines (Ollama, llama.cpp), this sets the actual context-window memory the model allocates at runtime.

ALLOCATED CONTEXT = ALLOCATED VRAMLlama 3.1 8B · KV cache footprintnum_ctx=2K0.27 GBnum_ctx=8K1.05 GBnum_ctx=32K4.20 GBnum_ctx=128K16.80 GBOllama defaults to 2048 — bump it explicitly for long documents
Definition

context-window is the model's theoretical max (Llama 3.1 8B = 128K). num_ctx is the memory you actually allocate at runtime. Engines often keep it below the model's ceiling because context size scales VRAM linearly.

Math: KV cache ≈ 2 (K+V) × num_layers × num_kv_heads × head_dim × bytes × num_ctx. On Llama 3.1 8B (32 layers, 8 KV heads, head_dim 128, f16) that's ~131 KB per context token. 8K context → ~1 GB VRAM. 32K → ~4 GB. 128K → ~16 GB just for KV cache — on top of model weights.

Provider variations (as of October 2026):

  • Ollama: OLLAMA_CONTEXT_LENGTH=8192 env or PARAMETER num_ctx 8192 in the Modelfile. The default is picked from VRAM: under 24 GiB → 4096, 24–48 GiB → 32768, 48 GiB and up → 262144. If you ask for more than the model was trained on, Ollama clamps it down.
  • llama.cpp: -c / --ctx-size. Default 0 = the model's trained context. --fit (on by default) may lower it, down to --fit-ctx (default 4096), if it doesn't fit in memory.
  • vLLM: --max-model-len 8192. If unset, derived from the model config.
  • OpenAI/Anthropic: not exposed — API handles it.
Analogy

Setting how many pages a notebook has. The model's "memory capacity" is finite — you reserve pages up front. 4 pages (num_ctx=4096) = fast but forgetful. 100 pages (128K) = takes lots of desk space (VRAM) but remembers long conversations.

Real-world example

Running Llama 3.1 8B on Ollama for chat on a 16 GB Mac:

`` ollama run llama3.1:8b > Pasting a 50K-token doc... > [model only "sees" the last ~4K tokens] ``

Cause: below 24 GiB of VRAM, num_ctx defaults to 4096. Ollama trims an oversized prompt from the front, so the start of the doc is ignored.

Fix: start with OLLAMA_CONTEXT_LENGTH=32768 ollama serve. Or Modelfile: `` FROM llama3.1:8b PARAMETER num_ctx 32768 ` Now the last ~32K of the 50K doc is kept; the first ~18K still drops. Check what was actually allocated in the CONTEXT column of ollama ps`.

VRAM (approximate): num_ctx=4096 → ~5-6 GB total (8B Q4 + cache). num_ctx=32768 → ~9 GB. Fits comfortably on a 16 GB Mac.

When to use
  • RAG or long-doc analysis — the 4K default is not enough
  • Multi-turn chat — 8K+ to keep history
  • Code editing and agent tasks — Ollama's docs recommend at least 64000 tokens
  • VRAM headroom available — go ahead, set 32K-128K
When not to use
  • One-shot short queries — 2K-4K is fine, save VRAM
  • Tight VRAM (8 GB Mac) with a big model (13B+) — trade off num_ctx vs model size
  • Beyond model's true max — Llama 3.1 caps at 128K; Ollama clamps larger values to 128K with a warning
Common pitfalls

The low-default surprise

Llama 3.1 says 'supports 128K' but on a machine under 24 GiB of VRAM Ollama defaults to 4096! For long context, set it explicitly. Note: Ollama's Modelfile page still says 2048 and its FAQ says 4096; current behaviour is the VRAM-based tiers.

Skipping the VRAM math

num_ctx=128K, 8B Q4 model, 16 GB Mac → OOM. Compute first: model + (num_layers × kv_heads × head_dim × 4 × num_ctx). Weight quantization (Q4) doesn't shrink the cache; quantize the cache separately (OLLAMA_KV_CACHE_TYPE=q8_0, or -ctk/-ctv in llama.cpp), see Flash Attention.

Pushing past the model's real ceiling

'128K supported' doesn't mean it produces good output at 128K. Lost-in-the-middle is real. Cap at 32K and use RAG to manage the window — better results.