How to Run an LLM Locally (2026): Hardware, Runtimes, Models and Setup

Summary

A "local LLM" means running a large language model (LLM) on your own computer or server instead of calling a cloud API. As of 2026 the setup is a handful of commands: install a runtime (for most people Ollama), download an open-weight model, talk to it through an OpenAI-compatible API.

The hard part is not installation but sizing: which model fits your memory, which quantization level keeps quality acceptable, and where memory goes as the context window grows. This guide answers those in order, then walks through setup, a UI, moving to a server, performance tuning and troubleshooting.

Note: Commands, default ports and environment variables were checked on October 24, 2026 against the official docs and GitHub READMEs (latest releases on that date: Ollama v0.40.10, vLLM v0.32.0, Open WebUI v0.12.3, mlx-lm v0.32.2). Model sizes come from the tag pages of the Ollama library. These tools move fast; if a command fails, check the official docs first.

Each runtime already has a detailed page under Local AI tools. We don't repeat them here; we connect them.

Why run it locally?

• Privacy and data sovereignty: Prompts, documents and customer data never leave the machine. For teams handling personal data this is the most concrete benefit under GDPR and Turkey's KVKK: sending data to an API provider abroad can trigger the GDPR's international-transfer rules (Chapter V) or KVKK's cross-border transfer provisions (Art. 9). With a local setup that transfer never happens. (This is not legal advice; ask a specialist about your case.)
• Predictable cost: No per-token billing. For heavy, repetitive workloads (batch classification, generating embeddings, internal search) you pay for hardware once. The cost calculator helps you compare with the cloud.
• Offline use: Works on a plane, in the field, on restricted networks.
• Control: The model version doesn't change unless you change it. System prompt, sampling and context length are entirely yours.
• Learning: Local is the best place to see how inference really works and why the KV cache eats memory.

There are costs too: hardware, maintenance, security and lower quality than the best cloud models on the hardest tasks. The "When not to self-host" section weighs that honestly.

Hardware: do the memory math

The deciding resource for a local LLM is memory: VRAM on NVIDIA/AMD cards, unified memory on Apple Silicon, system RAM on CPU-only machines. If the model doesn't fit, it either won't load or spills partly to the CPU and slows down sharply.

The formula

Total memory ≈ weights + KV cache + runtime overhead

weights  ≈ parameter count × (bits per weight ÷ 8)
KV cache ≈ 2 × layers × KV heads × head_dim × bytes × tokens × concurrent requests

The bits-per-weight values come from llama.cpp's quantize README, which has an official "bits/weight" table measured on Llama 3.1 8B (Q4_K_M 4.89, Q5_K_M 5.70, Q6_K 6.56, Q8_0 8.50, Q3_K_M 4.00). The same README lists the Q4_K_M file for Llama 3.1 70B at 43.1 GB, and Ollama's llama3.3:70b (q4_K_M) tag is 43 GB. The formula matches real files.

Bits per weight vary slightly by model (K-quants keep some tensors at higher precision). The table covers weights only, rounded. For an exact estimate use the site's GPU / VRAM calculator.

Don't forget the KV cache

The KV cache holds key/value tensors for every token the model has seen, and it grows linearly with context length. Example: according to the official architecture definition in Meta's llama-models repo, Llama 3.3 70B has 80 layers, 64 attention heads, 8 KV heads and a hidden size of 8192, so the head dimension is 8192 ÷ 64 = 128. At F16, per token:

2 × 80 × 8 × 128 × 2 bytes = 327,680 bytes ≈ 0.31 MiB / token
32,768 tokens  → ~10 GiB
131,072 tokens → ~40 GiB

So running a 43 GB model at 128K context can need roughly as much memory again. Dropping the KV cache to 8-bit (q8_0) roughly halves it, according to the Ollama docs. Newer architectures with sliding-window or hybrid attention (Gemma 4's model card, for instance, lists a 1024-token sliding window) can use less; check the model card. There's also a context calculator for planning.

Which hardware?

Per Ollama's GPU docs, NVIDIA needs compute capability 5.0+ and driver 550+; Apple devices are accelerated through Metal.

Real numbers: the llama.cpp repo has two discussions where the community runs the same test (Llama 2 7B, Q4_0, llama-bench) on different hardware: Apple Silicon (#4167) and NVIDIA CUDA (#15013). A few rows (PP = prompt processing, TG = token generation, tokens/second):

CUDA numbers are from the table without Flash Attention. These are community measurements taken on different llama.cpp versions, with a small 7B model that fits on every card; with large models the picture changes (a model that doesn't fit in 24 GB spills to the CPU on an NVIDIA card but not on a high-memory Mac). Still, two trends are clear: NVIDIA cards are far ahead on prompt processing, while the gap on token generation is smaller and tracks bandwidth.

A rule of thumb: token generation (decode) is mostly memory-bandwidth bound. A rough upper bound is bandwidth ÷ bytes of weights read per token. That's why Mixture of Experts (MoE) models are so attractive locally: the total parameter count is large, but each token only reads the active experts. The throughput calculator estimates speed for your hardware. It's an estimate; real numbers depend on runtime, quantization and context length.

Choosing a runtime

The runtime is the software that loads the model file and generates tokens. Each has a detailed page under Local AI; in short:

Practical decision:

• First time: Ollama or LM Studio.
• On a Mac and want every last bit: MLX (Ollama and LM Studio also offer -mlx tagged / MLX-engine models).
• Serving 5+ people from a GPU server: vLLM.
• Unusual hardware, a CPU-only server, or you want every flag: llama.cpp.

Because Ollama and LM Studio use llama.cpp-family engines under the hood, GGUF files and quantization names (Q4_K_M etc.) are shared across all three.

Choosing a model

As of October 2026, these families are available in the Ollama library with tags we checked (sizes are for Ollama's default quantization):

Selection tips:

1. Pick what fits in ~70–80% of your memory and leave the rest for the KV cache and the OS. On a 16 GB card, 8–14B Q4 models or gemma4:12b are comfortable; on 24 GB, 27–35B Q4 models fit but room for long context gets tight.
2. Look at MoE first: models like gemma4:26b, qwen3.6:35b-a3b and qwen3-coder:30b need memory for their full size but read far fewer parameters per token, so they're usually faster than dense equivalents in the same memory.
3. Match the task: if you need tool calling, pick a model with the tools badge on Ollama; for image input, vision.
4. Start small: 1–4B small language models are surprisingly good at classification, extraction and routing. More in the small language models guide.
5. Read the license: "open-weight" doesn't always mean "free for any use"; check the license on the model card before commercial use.

For hardware-based suggestions use the model recommender; to see models side by side, the comparison tool.

Quickstart: Ollama in 10 minutes

1. Install

# macOS and Linux
curl -fsSL https://ollama.com/install.sh | sh

# Windows (PowerShell)
irm https://ollama.com/install.ps1 | iex

If you prefer Docker (with an NVIDIA GPU):

docker run -d --gpus=all -v ollama:/root/.ollama \
  -p 127.0.0.1:11434:11434 --name ollama ollama/ollama

The official docs use -p 11434:11434, which publishes the port on every interface. The 127.0.0.1: prefix above binds it to the local machine only.

2. Pull a model and chat

ollama pull gemma4:e4b
ollama run gemma4:e4b

ollama run opens an interactive chat; /bye exits. Other useful commands:

ollama ls          # downloaded models
ollama ps          # loaded models, GPU/CPU split and context size
ollama stop gemma4:e4b   # unload a model from memory immediately

3. Call the OpenAI-compatible API

Ollama serves OpenAI-compatible endpoints under http://localhost:11434/v1/ (/v1/chat/completions, /v1/completions, /v1/models, /v1/embeddings, /v1/responses …).

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemma4:e4b",
    "messages": [
      {"role": "system", "content": "Answer briefly and clearly."},
      {"role": "user", "content": "What is a KV cache? Explain in two sentences."}
    ]
  }'

In Python with the official openai package:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1/",
    api_key="ollama",  # required but ignored by Ollama
)

resp = client.chat.completions.create(
    model="gemma4:e4b",
    messages=[{"role": "user", "content": "Explain quantization in one sentence."}],
)
print(resp.choices[0].message.content)

The same code works against LM Studio (http://localhost:1234/v1), vLLM (http://localhost:8000/v1) or the mlx-lm server (http://localhost:8080/v1) by changing only base_url and the model name. It's the shortest path for moving an existing cloud-API app to local.

For Ollama-specific options (for example num_ctx), use the native API:

curl http://localhost:11434/api/chat -d '{
  "model": "gemma4:e4b",
  "messages": [{"role": "user", "content": "Hello"}],
  "options": {"num_ctx": 16384},
  "stream": false
}'

The same step with other runtimes

# llama.cpp: downloads a GGUF from Hugging Face and starts an OpenAI-compatible server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

# vLLM: serves a Hugging Face-format model on port 8000
uv pip install vllm --torch-backend=auto
vllm serve Qwen/Qwen2.5-1.5B-Instruct

# MLX (Apple Silicon)
pip install mlx-lm
mlx_lm.server --model mlx-community/Mistral-7B-Instruct-v0.3-4bit

# Start the server with the LM Studio CLI
lms server start

llama.cpp's current README installs via the llama.app script and shows a single llama command (llama cli, llama serve); the separate llama-server binary is still built too, so the ./llama-server -m model.gguf form you'll see in older guides still works.

Connecting a UI and coding tools

Open WebUI

Open WebUI is a self-hosted, multi-user, ChatGPT-style interface. It connects to Ollama and any OpenAI-compatible server. The Docker command from its README (with Ollama on the same machine):

docker run -d -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data \
  --name open-webui --restart always \
  ghcr.io/open-webui/open-webui:main

Then open http://localhost:3000. If Ollama runs on another machine, add -e OLLAMA_BASE_URL=http://server-address:11434. Without Docker: pip install open-webui and open-webui serve (defaults to http://localhost:8080).

If the container can't reach Ollama, the README suggests --network=host; the address then becomes http://localhost:8080.

Coding tools

• Ollama integrations: The current Ollama README shows the ollama launch command for wiring tools such as Claude Code, Codex, Copilot CLI, OpenCode and Droid to local or cloud models. For example, ollama launch claude runs Claude Code through Ollama's Anthropic-compatible API.
• Any OpenAI-compatible tool: Editor extensions and CLIs with a "custom OpenAI endpoint" or "base URL" setting work with http://localhost:11434/v1.
• Context for agent workloads: Ollama's docs recommend at least 64,000 tokens of context for web search, agents and coding tools. Leave the default (see below) and your agent will "forget" files.
• Adding tools: You can give a local model your own tools through MCP; step by step: Build your own MCP server. To chat with your documents, set up RAG.

Keep in mind that small local models make noticeably more mistakes than large cloud models on long, multi-step agent tasks. For coding, the best local results usually come from 27B+ coding/agent-focused models (e.g. qwen3-coder:30b, qwen3.6:27b-coding); that's a general observation, so test on your own repos.

Serving on a server or LAN (securely)

By default Ollama listens only on 127.0.0.1:11434. That's the safe default. To reach it from other devices you need the OLLAMA_HOST variable.

Linux (systemd)

sudo systemctl edit ollama

In the editor:

[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="OLLAMA_CONTEXT_LENGTH=32768"
Environment="OLLAMA_KEEP_ALIVE=30m"

Then:

sudo systemctl daemon-reload
sudo systemctl restart ollama
journalctl -e -u ollama   # logs

For the macOS app, set variables with launchctl setenv OLLAMA_HOST "0.0.0.0:11434" and restart the app; on Windows, add them as user environment variables. Models live in ~/.ollama/models on macOS and /usr/share/ollama/.ollama/models on Linux by default; point OLLAMA_MODELS elsewhere if you run out of disk.

Security rules

1. Never expose an unauthenticated port to the internet. Ollama's docs describe OLLAMA_HOST, reverse proxies such as Nginx, and tunnels such as ngrok or Cloudflare Tunnel for exposing it; they define no API-key setting for the local server. The source does have an OLLAMA_AUTH variable, but in api/client.go it's used client-side to sign requests (the same path used for ollama.com); it isn't documented as a way to password-protect the local API. The GitHub issue requesting API-key and TLS support (#14003) is still open as of October 2026. You have to add authentication at the proxy layer. An open LLM endpoint means strangers using your GPU for free, and deleting or pulling models.
2. Prefer a specific interface over 0.0.0.0: bind to your LAN IP or a VPN interface (e.g. Tailscale/WireGuard). Open 11434 in the firewall only to networks you trust.
3. Careful when publishing Docker ports: -p 11434:11434 listens on all interfaces. For local-only access use -p 127.0.0.1:11434:11434.
4. Servers with key support: the llama.cpp server supports --api-key (or LLAMA_API_KEY), vLLM supports --api-key (or VLLM_API_KEY). Still put a proxy in front for TLS.
5. Lock down the UI too: if you expose an interface like Open WebUI, keep user sign-up under control and use HTTPS.
6. Logs contain personal data: set retention and access for prompt/response logs according to your GDPR/KVKK obligations. Running locally doesn't remove your data privacy responsibility; it moves it to you.
7. Prompt injection exists locally too: if the model has tool and file access, untrusted content can still hijack its behavior.

Example Nginx proxy config (from the Ollama FAQ; you must add auth and TLS yourself):

server {
    listen 80;
    server_name example.com;
    location / {
        proxy_pass http://localhost:11434;
        proxy_set_header Host localhost:11434;
    }
}

Concurrency on a small server

• OLLAMA_NUM_PARALLEL: requests processed at once per model (default 1). Required memory scales with OLLAMA_NUM_PARALLEL × OLLAMA_CONTEXT_LENGTH.
• OLLAMA_MAX_LOADED_MODELS: models loaded at once (default 3 × number of GPUs, or 3 on CPU).
• OLLAMA_MAX_QUEUE: how many requests are queued when busy before rejecting (default 512).

If you'll serve several users steadily, vLLM is built for it (continuous batching, PagedAttention). Details: /local-ai/vllm.

Performance tuning

Quantization level

Quantization stores weights in fewer bits. Common practice: Q4_K_M is the safe default for the memory/quality trade-off; with memory to spare, Q5/Q6 or Q8_0 add a little quality; Q3 and below can lose noticeable quality on small models. llama.cpp's own llama-quantize list gives the perplexity increase over F16 on Llama-3-8B as: Q8_0 +0.003, Q6_K +0.022, Q5_K_M +0.057, Q4_K_M +0.175, Q3_K_M +0.657, Q2_K +3.52. The loss grows quickly below Q4. When memory is tight, a bigger model at lower bits often beats a smaller model at higher bits, but measure it on your task. In Ollama you pick quantization by tag (llama3.3:70b-instruct-q4_K_M); in llama.cpp with -hf repo:Q5_K_M (defaults to Q4_K_M if omitted).

Context length (num_ctx)

num_ctx is how many tokens the model can see in one request. According to Ollama's context-length docs the default depends on VRAM: 4K under 24 GiB, 32K for 24–48 GiB, 256K at 48 GiB and above. If you work with long documents or agents, raise it deliberately:

# Server-wide
OLLAMA_CONTEXT_LENGTH=32768 ollama serve

# Inside a chat
/set parameter num_ctx 32768

Or create a persistent model variant:

# Modelfile
FROM gemma4:e4b
PARAMETER num_ctx 32768

ollama create gemma4-32k -f Modelfile

Check the context actually allocated in the CONTEXT column of ollama ps. Larger context means a larger KV cache (see the math above); once the model starts spilling off the GPU, speed drops.

Layers on the GPU (n_gpu_layers)

n_gpu_layers is how many of the model's layers stay on the GPU. In llama.cpp it's -ngl (default auto; also accepts all or a number). Ollama computes it automatically; in ollama ps, 100% GPU means fully on the GPU, while something like 48%/52% CPU/GPU means a partial spill. If you see a spill, shrink the context or pick a smaller quantization first; only then play with layer counts.

Batch size

Batch size is how many tokens are processed at once during prompt processing (prefill). In llama.cpp it's -b (logical, default 2048) and -ub (physical, default 512). Bigger values can speed up long prompts but cost memory. The Ollama equivalent is the num_batch option.

KV cache and Flash Attention

• Flash Attention: Ollama enables it automatically where the backend supports it; force it with OLLAMA_FLASH_ATTENTION=1. In llama.cpp, -fa auto|on|off (default auto).
• KV cache quantization: in Ollama, OLLAMA_KV_CACHE_TYPE=q8_0 (~1/2 memory, quality loss usually unnoticeable) or q4_0 (~1/4, loss can be more noticeable at long context). Default is f16, and the setting is global across models. In llama.cpp, -ctk q8_0 -ctv q8_0. The MLX server has --kv-bits, but per the mlx-lm docs a quantized KV cache doesn't support batching.

Keep the model loaded

By default Ollama unloads a model after 5 minutes idle, which is why the first request is slow. Change it with OLLAMA_KEEP_ALIVE=30m (or -1 for indefinitely) or the per-request "keep_alive" parameter.

Example: a fully tuned llama.cpp server

llama-server -m ./models/model-Q4_K_M.gguf \
  -c 32768 -ngl all -fa on \
  -ctk q8_0 -ctv q8_0 \
  -b 2048 -ub 512 -np 2 \
  --host 127.0.0.1 --port 8080 \
  --api-key "$LLAMA_API_KEY"

Measure which setting matters on your own hardware: change one variable at a time and watch tokens/second. Cutting tokens from long prompts is also the cheapest speed-up: Token reduction techniques.

Troubleshooting

On Linux, logs: journalctl -e -u ollama.

When not to self-host

• You need top quality on the hardest tasks: long agent loops, complex multi-file coding, hard reasoning. Models that fit consumer hardware are usually behind the best cloud models here. Compare on your own test set.
• Usage is low and irregular: for a few thousand requests a month, hardware and electricity can cost more than the API bill. Compare with the cost calculator.
• You have no ops capacity: updates, monitoring, backups, security patches and outages become your job.
• You need compliance certification: in some sectors an audited, contracted cloud provider (data processing agreement, regional data center) is easier to document than a server you built.
• Load spikes unpredictably: one GPU server can't absorb a sudden 50× spike; the cloud can.

For most teams the best answer is hybrid: sensitive data and high-volume routine work local, the hardest and rarest work in the cloud. With the same OpenAI-compatible client you switch between them by changing only base_url.

Checklist

• Do model + KV cache fit in ~80% of memory? (VRAM calculator)
• Does ollama ps show 100% GPU?
• Is context length set deliberately for the workload?
• Is the API only on 127.0.0.1, or behind an authenticated proxy?
• Is a retention period set for prompt/response logs?
• Does the model's license fit your use?
• Have you compared against a cloud model on your own tasks?

Further reading

• Local AI tools: detailed pages for Ollama, llama.cpp, LM Studio, vLLM and MLX.
• Quantization, KV Cache, num_ctx, n_gpu_layers, Batch Size — short definitions of the tuning knobs.
• Mixture of Experts — why big models can be fast locally.
• Small language models guide — what you can do with 1–4B models.
• Tools: VRAM calculator, throughput, model recommender, context.

Sources

• Ollama README and install: https://github.com/ollama/ollama
• Ollama FAQ (OLLAMA_HOST, keep_alive, concurrency, KV cache, proxies): https://docs.ollama.com/faq
• Ollama context length: https://docs.ollama.com/context-length
• Ollama OpenAI compatibility: https://docs.ollama.com/api/openai-compatibility
• Ollama GPU support: https://docs.ollama.com/gpu
• Ollama Linux and Docker: https://docs.ollama.com/linux, https://docs.ollama.com/docker
• Ollama Claude Code integration: https://docs.ollama.com/integrations/claude-code
• Ollama model library and tag pages: https://ollama.com/library
• llama.cpp README and server docs: https://github.com/ggml-org/llama.cpp, https://github.com/ggml-org/llama.cpp/tree/master/tools/server
• vLLM quickstart: https://docs.vllm.ai/en/stable/getting_started/quickstart/
• mlx-lm README and server docs: https://github.com/ml-explore/mlx-lm
• LM Studio local server, CLI and OpenAI compatibility: https://lmstudio.ai/docs/developer/core/server, https://lmstudio.ai/docs/cli
• Open WebUI README: https://github.com/open-webui/open-webui
• Llama 3.3 70B architecture definition, Meta llama-models (KV cache math): https://github.com/meta-llama/llama-models/blob/main/models/sku_list.py
• llama.cpp quantize README (bits/weight table, Q4_K_M file sizes): https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md
• llama.cpp llama-quantize perplexity list: https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/quantize.cpp
• llama.cpp Apple Silicon performance discussion: https://github.com/ggml-org/llama.cpp/discussions/4167
• llama.cpp NVIDIA CUDA performance discussion: https://github.com/ggml-org/llama.cpp/discussions/15013
• Ollama client code (OLLAMA_AUTH): https://github.com/ollama/ollama/blob/main/api/client.go
• Ollama issue #14003 (API key and TLS request): https://github.com/ollama/ollama/issues/14003

Frequently asked questions

How much VRAM do I need to run an LLM locally?

Roughly parameter count × bytes per weight, plus the KV cache. For a 4-bit (Q4_K_M) model budget about 0.6 GB per billion parameters: an 8B model is ~5 GB, 32B ~20 GB, 70B ~43 GB, plus a few GB of KV cache depending on context length.

Is Ollama free?

Yes. Ollama is open-source software under the MIT license and running it locally costs nothing. Ollama also offers optional cloud models, which have their own terms.

Can I run an LLM without a GPU?

Yes, llama.cpp and Ollama both run on CPU only. Speed depends mostly on RAM bandwidth: small 1-4B models or MoE models with few active parameters are usable on CPU, while large dense models are very slow.

Is a local LLM as good as ChatGPT or Claude?

Models that fit on consumer hardware are good enough for many everyday tasks, but large cloud models are usually still ahead on the hardest reasoning and long agentic tasks. The reliable way to know is a small test set built from your own tasks.

Ollama vs LM Studio vs llama.cpp: which should I use?

Ollama for a fast terminal and API-first start, LM Studio for a graphical app and model discovery, llama.cpp for the most control and the widest hardware support. Use vLLM for a multi-user GPU server and MLX for the most native performance on a Mac.

Is a local LLM better for GDPR compliance?

Because data never leaves your infrastructure, international-transfer and third-party processor risks shrink. Your obligations (access control, logging, retention) still remain, and a badly locked-down server can leak data just as easily.