Running LLMs locally — hardware, tools, models
How to run language models on your own hardware — VRAM requirements, tooling (Ollama, LM Studio, llama.cpp, vLLM) and which models fit which GPU.
Martin Rau
Local LLM Concepts
in AI Concepts
Konzepte rund um lokalen Modellbetrieb — Formate, Quantisierung, Hardware.
Glossary
GGUF Lokales LLM
GGUF (GPT-Generated Unified Format) is the standard file format for quantised LLMs on the llama.cpp engine — a single `.gguf` file holds weights, tokenizer and metadata together.
Quantization Lokales LLM
Quantization reduces the numeric precision of model weights — e.g. from 16-bit float to 4-bit integer — making language models small and fast enough for consumer hardware, with manageable quality loss.
VRAM Lokales LLM
VRAM (Video RAM) is the dedicated memory on a GPU — the single most important hardware figure for running language models locally, because ideally the whole model and its context must fit inside it.
Batch Inference Lokales LLM
Batch inference is the bundled processing of many prompts in one pass — locally on a GPU or as an asynchronous API job — trading real-time latency for throughput and cost.