num_ctx
Runtime context length parameter
On local inference engines (Ollama, llama.cpp), this sets the actual context-window memory the model allocates at runtime.
context-window is the model's theoretical max (Llama 3.1 8B = 128K).
num_ctx is the memory you actually allocate at runtime. Engines
often keep it below the model's ceiling because context size scales
VRAM linearly.
Math: KV cache ≈ 2 (K+V) × num_layers × num_kv_heads × head_dim × bytes × num_ctx. On Llama 3.1 8B (32 layers, 8 KV heads, head_dim 128, f16) that's ~131 KB per context token. 8K context → ~1 GB VRAM. 32K → ~4 GB. 128K → ~16 GB just for KV cache — on top of model weights.
Provider variations (as of October 2026):
- Ollama:
OLLAMA_CONTEXT_LENGTH=8192env orPARAMETER num_ctx 8192in the Modelfile. The default is picked from VRAM: under 24 GiB → 4096, 24–48 GiB → 32768, 48 GiB and up → 262144. If you ask for more than the model was trained on, Ollama clamps it down. - llama.cpp:
-c/--ctx-size. Default0= the model's trained context.--fit(on by default) may lower it, down to--fit-ctx(default 4096), if it doesn't fit in memory. - vLLM:
--max-model-len 8192. If unset, derived from the model config. - OpenAI/Anthropic: not exposed — API handles it.
Setting how many pages a notebook has. The model's "memory capacity" is finite — you reserve pages up front. 4 pages (num_ctx=4096) = fast but forgetful. 100 pages (128K) = takes lots of desk space (VRAM) but remembers long conversations.
Running Llama 3.1 8B on Ollama for chat on a 16 GB Mac:
``
ollama run llama3.1:8b
> Pasting a 50K-token doc...
> [model only "sees" the last ~4K tokens]
``
Cause: below 24 GiB of VRAM, num_ctx defaults to 4096. Ollama trims
an oversized prompt from the front, so the start of the doc is ignored.
Fix: start with OLLAMA_CONTEXT_LENGTH=32768 ollama serve. Or
Modelfile:
``
FROM llama3.1:8b
PARAMETER num_ctx 32768
`
Now the last ~32K of the 50K doc is kept; the first ~18K still drops.
Check what was actually allocated in the CONTEXT column of
ollama ps`.
VRAM (approximate): num_ctx=4096 → ~5-6 GB total (8B Q4 + cache). num_ctx=32768 → ~9 GB. Fits comfortably on a 16 GB Mac.
- RAG or long-doc analysis — the 4K default is not enough
- Multi-turn chat — 8K+ to keep history
- Code editing and agent tasks — Ollama's docs recommend at least 64000 tokens
- VRAM headroom available — go ahead, set 32K-128K
- One-shot short queries — 2K-4K is fine, save VRAM
- Tight VRAM (8 GB Mac) with a big model (13B+) — trade off num_ctx vs model size
- Beyond model's true max — Llama 3.1 caps at 128K; Ollama clamps larger values to 128K with a warning
The low-default surprise
Llama 3.1 says 'supports 128K' but on a machine under 24 GiB of VRAM Ollama defaults to 4096! For long context, set it explicitly. Note: Ollama's Modelfile page still says 2048 and its FAQ says 4096; current behaviour is the VRAM-based tiers.
Skipping the VRAM math
num_ctx=128K, 8B Q4 model, 16 GB Mac → OOM. Compute first: model + (num_layers × kv_heads × head_dim × 4 × num_ctx). Weight quantization (Q4) doesn't shrink the cache; quantize the cache separately (OLLAMA_KV_CACHE_TYPE=q8_0, or -ctk/-ctv in llama.cpp), see Flash Attention.
Pushing past the model's real ceiling
'128K supported' doesn't mean it produces good output at 128K. Lost-in-the-middle is real. Cap at 32K and use RAG to manage the window — better results.