num_thread & llama.cpp --threads: How Many?

Definition

When some or all of the model runs on the CPU (partial offload via n_gpu_layers, or a machine without a GPU), the matrix multiplications are split across threads. The thread count directly affects speed, but "more is better" doesn't hold.

llama.cpp has two separate settings:

-t / --threads: threads for token generation.
-tb / --threads-batch: threads for prompt (batch) processing. Default: same as --threads.

Without -t (default -1, as of October 2026) llama.cpp picks the number of cores "useful for math" itself:

macOS: the performance-core count (hw.perflevel0.physicalcpu), efficiency cores excluded.
Linux x86-64, hybrid CPU: P-cores only, E-cores and hyperthread siblings excluded. The reasoning in the source: E-cores slow down lockstep threading, and hyperthreading doesn't help linear algebra.
Other Linux / Windows: the physical core count.

Ollama: num_thread defaults to 0 = "let the runtime decide". Ollama runs GGUF models through the llama.cpp server and passes num_thread as -t only when it's greater than 0; otherwise the llama.cpp choice above applies. It's no longer listed in the current Modelfile parameter table but remains available as options.num_thread in the API.

Analogy

Cooks preparing the same dish. One cook per stove (physical core) is efficient. A second cook at the same stove (hyperthread) doesn't speed things up; they get in each other's way. Adding apprentices (E-cores) also hurts: everyone waits for the slowest one to finish.

Real-world example

A Linux desktop with 8 P-cores + 8 E-cores, showing 24 logical threads. The model runs entirely on the CPU.

Setting	What happens
`-t 24`	Every logical thread; E-cores and hyperthreads become the bottleneck
`-t 16`	Mixed P + E; the slowest core sets the pace
`-t 8`	P-cores only; what llama.cpp picks by default

Which one is fastest depends on the hardware; measure, don't guess. llama.cpp's llama-bench tries several values in one run:

``llama-bench -m model.gguf -t 4,6,8,12,16``

Code examples

                                            Ollama API · setting the thread count by hand
                                            bash
                                        
curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Hello",
  "options": { "num_thread": 8 }
}'

                                            llama.cpp · separate thread counts for generation and prompt processing
                                            bash
                                        
# New install (llama.app): llama serve
# Older binary name: llama-server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF \
  --threads 8 \
  --threads-batch 16 \
  -ngl 0
# The "system_info: n_threads = ..." line in the startup log
# shows the value actually used

When to use

The model runs fully or partly on the CPU — one of the most impactful speed knobs
Other heavy work runs on the machine — lower it to leave a few cores free
Containers or VMs with a CPU limit — auto-detection may see the host's cores
A benchmark points to a value other than the default

When not to use

The model is fully on the GPU — compute happens there, the thread count has limited effect
Auto-detection is already right — the physical core count is a good starting point
To 'match the logical thread count' — that usually makes it slower

Common pitfalls

Using the logical core count

The number your task manager shows (hyperthreads included) can be twice the physical cores. Token generation is bound by memory bandwidth, so extra threads usually bring no gain and can slow things down through contention.

The memory-bandwidth ceiling

Past a certain point, more threads don't raise tokens/s; RAM bandwidth is the bottleneck. If 6 and 12 threads give the same result, pick the lower one and leave the rest of the machine some room.

Container quotas

With a Docker quota like --cpus 4, auto-detection may still count all of the host's cores. Set -t (or num_thread) to match the quota explicitly.