Term
Inference
Inference is the act of running an already-trained model to turn an input into a response — production use, not training.
Inference — explained in more detail
Where training fits a model to data, inference is the process of using the finished model: a prompt comes in, tokens are computed one after another, and the output emerges. Inference is far cheaper than training but adds up quickly in production — every request consumes compute (GPU/CPU), memory, and energy.
Example / Practical context
A local Llama 3 model on a laptop runs in inference mode: llama.cpp loads the weights into RAM, the prompt is pushed through the network, tokens are sampled. With API services like the Anthropic or OpenAI API, users pay per inference, billed by input and output tokens. Knobs such as quantization, batch size, and KV-cache determine speed and cost.
Distinction from related concepts
Inference is neither training nor fine-tuning — the weights do not change during it. It is also not a simple single-class prediction: in generative models, inference is an iterative process that builds an entire output token by token.
Discover more
AI Workflows by Keyword: How We Make Recurring Routines Enforceable
A typed keyword triggers a fixed AI routine — and every single step must be committed before the next one appears. Why that's the actual trick.
GlossaryAI Watermarking
A method AI providers use to mark generated content so machines can recognise it. In text this is an invisible statistical pattern in word choice; in files it is signed provenance metadata. The goal is evidence that a model was involved in producing the content.
EncyclopediaLLM Hallucinations — Causes and Remedies
Why LLMs confidently invent falsehoods, what types of hallucinations exist, and which remedies actually help — RAG, source enforcement, verification.