LLM-as-a-Judge

Properties2

Type	Concept
Note created	Mar 20, 2025

LLM-as-a-judge is a practice that uses a large language model to evaluate a given input, mimicking what an expert evaluation would provide. That way, LLMs are used as substitutes for human evaluation in tasks, serving as some sort of vague error metric.

There is no forma definition of LLM-as-a-judge, and some examples of it in use are:

• Evaluating how similar, correct or complete a text is based on a reference text,
• Using an LLM considered “good enough” to evaluate the task of a different LLM by performing the task itself (with no ground truth for reference).