Advanced · ~2 min read #flash-attention #kv-cache #inference

Flash Attention

Memory-efficient attention and -fa / OLLAMA_FLASH_ATTENTION

An algorithm that computes attention in small blocks without ever writing the N×N score matrix to GPU memory. Same result, far less memory traffic. In local inference, a prerequisite for KV cache quantization.

Definition

Standard attention computes a score for every pair of tokens and writes that N×N matrix to the GPU's main memory (HBM), then reads it back. As context grows, memory and time grow quadratically.

FlashAttention (Dao et al., 2022, arXiv 2205.14135) produces the same result with no approximation, but splits the work into tiles that fit in the small, fast on-chip memory (SRAM). The full matrix is never materialized and reads/writes to HBM drop. Follow-ups: FlashAttention-2 (2023, better work partitioning, ~2× faster) and FlashAttention-3 (2024, Hopper GPUs and FP8).

In local engines (as of October 2026):

Engine	Setting	Default
llama.cpp	`-fa` / `--flash-attn` (on, off, auto)	`auto`
Ollama	`OLLAMA_FLASH_ATTENTION=1` / `=0`	Automatic when backend and device support it
llama.cpp KV cache	`-ctk` / `-ctv` (`--cache-type-k/-v`)	`f16`
Ollama KV cache	`OLLAMA_KV_CACHE_TYPE`	`f16`

llama.cpp doesn't use Dao's library here; it uses its own ggml kernel, FLASH_ATTN_EXT.

The KV cache dependency: in llama.cpp a quantized V cache does not work without Flash Attention. With -fa auto, llama.cpp turns FA on by itself; with -fa off, the model fails to load with "quantized V cache requires flash_attn". Ollama's docs also describe K/V cache quantization as working "when Flash Attention is enabled".

Analogy

Two ways to sum a huge table. One: write every intermediate result on a giant sheet in the warehouse (HBM), then walk back and read it. Two: add it up piece by piece in a small notebook on your desk (SRAM) and only carry the result to the warehouse. Same answer; the second way takes far fewer trips.

Real-world example

An 8B Q4 model on a 16 GB GPU, running with 32K context. The f16 KV cache alone takes ~4 GB (see num_ctx).

KV cache types, with the ratios from Ollama's docs:

`OLLAMA_KV_CACHE_TYPE`	Memory (vs f16)	Quality
`f16` (default)	1×	Full precision
`q8_0`	~1/2	Very small loss, usually not noticeable
`q4_0`	~1/4	Small-to-medium loss, can show at long context

With q8_0 the cache drops to ~2 GB; twice the context fits in the same VRAM.

``OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 ollama serve``

Code examples

                                            llama.cpp · Flash Attention + q8_0 KV cache
                                            bash
                                        
# New install (llama.app): llama serve
# Older binary name: llama-server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF \
  -c 32768 \
  -fa on \
  -ctk q8_0 \
  -ctv q8_0
# -fa off + quantized V cache = the model won't load

                                            Ollama · server-wide FA and KV cache type
                                            bash
                                        
# macOS app
launchctl setenv OLLAMA_FLASH_ATTENTION "1"
launchctl setenv OLLAMA_KV_CACHE_TYPE "q8_0"
# Restart Ollama. This is global: every model uses the q8_0 cache.

When to use

Long context (16K+) — where attention's memory and time cost hurts most
When you want to quantize the KV cache — required for a q8_0/q4_0 V cache
Trying to fit a num_ctx that doesn't fit in VRAM
On a supported GPU — in most cases auto already turns it on, nothing to do by hand

When not to use

The backend or model doesn't support it — e.g. llama.cpp forces FA off for the Grok architecture
You see wrong or slow output with FA in some version — compare with -fa off / OLLAMA_FLASH_ATTENTION=0
Simple jobs with short prompts — the gain stays small

Common pitfalls

Turning FA off and quantizing the cache

In llama.cpp, -fa off -ctv q8_0 stops the model from loading with 'quantized V cache requires flash_attn to be enabled'. In Ollama, combining OLLAMA_FLASH_ATTENTION=0 with OLLAMA_KV_CACHE_TYPE=q8_0 invites the same problem.

A global KV cache setting

OLLAMA_KV_CACHE_TYPE applies to every model. Per Ollama's docs, models with a high GQA count (e.g. Qwen2) may be hit harder by cache quantization. If a critical model loses quality, go back to f16.

Thinking Flash Attention shrinks the weights

FA doesn't touch the model weights; it cuts the memory traffic of the attention computation. Shrinking weights is quantization; shrinking the cache is the KV cache type setting.