Term
RAGAS
RAGAS is an open-source framework for automated evaluation of RAG and agent pipelines — with built-in metrics like faithfulness, context precision, and answer relevancy.
RAGAS — explained in detail
RAGAS (Retrieval-Augmented Generation Assessment) ships ready-made metrics to compare RAG systems objectively — without arranging a manual review for every change. Most metrics use LLM-as-a-Judge: a model checks, given the question, retrieved contexts, and final answer, whether the answer is actually grounded in the contexts (faithfulness) or whether the retriever delivered the relevant chunks (context precision/recall). It plugs into CI pipelines so each change is scored against an eval set automatically.
Example / Practical use
A common setup: build an eval set of question, gold answer, and gold contexts, run the pipeline, and let RAGAS compute per-example scores for faithfulness, answer relevancy, context precision, and context recall. Engineering teams use it to compare retrievers (vector vs. hybrid vs. reranked) or to gate prompt changes. Beyond RAG metrics, RAGAS also covers agent metrics (tool-call accuracy, goal achievement) and classical NLP scores (BLEU, ROUGE, semantic similarity).
Distinction from related terms
RAGAS is a specific library, not a concept — comparable to DeepEval, TruLens, and Promptfoo. The metrics themselves (faithfulness, context precision …) are universal and implemented in several frameworks. RAGAS is also not a benchmark: it is a measurement tool that runs on your own eval data. Weaknesses: LLM judges are expensive and slow, and can become unreliable in narrow domains — so production teams typically combine it with human spot checks.
Discover more
AI Workflows by Keyword: How We Make Recurring Routines Enforceable
A typed keyword triggers a fixed AI routine — and every single step must be committed before the next one appears. Why that's the actual trick.
GlossaryGround Truth
A reference taken to be correct, against which model outputs are measured. In AI code analysis it is a piece of code with deliberately injected, known defects, so you can measure how many real problems a model actually finds.
EncyclopediaAgent Evaluation — Task Success Rate, Eval Harnesses, Benchmarks
How to evaluate AI agents: eval harnesses, task success rate, benchmarks like GAIA and Tau-Bench, LLM-as-a-judge — and how it differs from pure model quality.