AI Atlas
Intermediate · ~2 min read #n_gpu_layers #gpu-offload #inference

n_gpu_layers

GPU layer offload count

A llama.cpp parameter that controls how many layers run on GPU vs CPU. A lifesaver when VRAM is tight.

HOW MANY LAYERS ON GPU vs CPUGPU · -ngl 5layer 1layer 2layer 3layer 4layer 5CPU + RAMlayer 6layer 7layer 8either all on GPU (-ngl 99) or hybrid — even one CPU layer is a bottleneck
Definition

llama.cpp (and Ollama, LM Studio built on top) can load model layers piecewise onto the GPU. -ngl N (long forms --gpu-layers or --n-gpu-layers): the first N layers run on GPU, the rest stay on CPU + RAM.

Practical values (as of October 2026):

  • Default auto: with --fit (on by default) llama.cpp tries to fit the model into free VRAM, leaving ~1 GiB headroom per device (tunable with --fit-target).
  • -ngl 0: all on CPU. Very slow but no VRAM for weights.
  • -ngl 20 (e.g.): first 20 of a 33-layer 8B model on GPU. Hybrid.
  • -ngl all or -ngl 99: all layers on GPU (fastest). Careful: -1 now means auto, not "all".

Speed scaling is non-linear. Full GPU = full speed; even one layer on CPU forces a CPU↔GPU transfer per token at that layer — major bottleneck. Rule of thumb: all or nothing is best; hybrid is acceptable for "make it work at all."

Math: VRAM per layer ≈ model size / num_layers. Llama 3.1 8B (Q4): ~5 GB / 33 layers ≈ 152 MB/layer. With 4 GB VRAM you can fit ~26 layers (-ngl 26); the remaining 7 stay on CPU. Approx speed: ~30-50% of full GPU.

Analogy

A restaurant kitchen. All cooks in the kitchen (GPU) → max speed. One cook in a side building (CPU) → every dish must travel through there for that step → the line jams. n_gpu_layers controls "how many cooks are in the main kitchen." Full or empty — anything in between is slow but possible.

Real-world example

You have an RTX 4080 with 16 GB VRAM. You want to run Llama 3.1 70B Q4 (~40 GB on disk, doesn't fit).

-ngl 20: 20 of 80 layers on GPU (~10 GB), 60 on CPU. Speed: ~3 tok/s. Slow but it runs.

-ngl 0: all CPU. Speed: ~0.8 tok/s. Works if you have 64 GB RAM.

Drop to a smaller model: 8B Q4 (~5 GB), -ngl all full GPU → ~140 tok/s. Same hardware, 47× speed difference.

By default Ollama doesn't pass -ngl at all and leaves the layer count to llama.cpp's auto mode. To set it by hand, send "options": {"num_gpu": 20} in the request (0 = CPU only). Check the result in the PROCESSOR column of ollama ps (e.g. 100% GPU). LM Studio's GUI has a "GPU Offload" slider for this.

When to use
  • VRAM is tight but you can't swap models — hybrid offload
  • Sharing one GPU across multiple models — budget each one
  • Quantization wasn't enough, still need some offload
  • Optimizing mixed CPU + GPU hardware
When not to use
  • Whole model fits on GPU — leave it on auto or set -ngl all and move on
  • High-throughput production — hybrid always tanks throughput
  • Tiny model (1-3B) — overhead not worth it
Common pitfalls

The GPU never gets used

The default is now auto, but if -ngl 0 (Ollama: num_gpu: 0) is left in place, or llama.cpp was built without GPU support, everything runs on CPU. In the second case llama.cpp prints 'no usable GPU found, --gpu-layers option will be ignored'. For 'why am I at 5 tok/s?', check the log and ollama ps first.

Forgetting the KV cache

The KV cache of each offloaded layer also lives in VRAM by default (llama.cpp --kv-offload is on; -nkvo keeps it in RAM). At 32K context the cache alone is ~4 GB. What must fit on the GPU: ngl × layer_size + those layers' cache.

Hybrid performance illusion

'50% GPU offload' ≠ 50% speed — usually worse. Each token incurs CPU↔GPU transfer latency. Benchmark, find 'X ngl gets Y tok/s' for your specific case.