Anyone who has tried to judge language model outputs knows the frustration: two reviewers read the same answer, one calls it excellent and the other calls it poor, and neither can explain exactly why. The disagreement is not a sign that one reviewer is wrong. It is a sign that the judgement was made without criteria. Human judgement without a rubric is noisy, and noisy judgement is a serious problem when you are choosing between models, testing a prompt change, or deciding whether an application is safe to release. A rubric fixes this by stating, before anyone judges anything, what good looks like, in enough detail that two reviewers applying it independently reach similar conclusions.
The LLM Evaluation Rubric Builder helps you construct that rubric: criteria, performance levels for each criterion, and descriptions of what each level means. The tool cannot decide what matters for your application; that is your judgement, and it is the most important part of the exercise. What the tool does is force you to write that judgement down precisely, which is where most evaluation efforts fail, because they skip from "we want good answers" to scoring, without ever defining good.
Start from the purpose, not from generic quality
The most common rubric mistake is copying a generic set of criteria, accuracy, fluency, helpfulness, and applying it to every task. Generic criteria sound reasonable and measure almost nothing, because they mean different things in different contexts. Accuracy for a medical summary means something very different from accuracy for a poem. A rubric should start from the purpose of the application: what is this output for, who will use it, and what would count as failure? A customer support answer fails if it gives the wrong procedure, even if it is beautifully written. A creative story fails if it is dull, even if it is grammatically perfect. A summary fails if it omits the crucial fact, even if everything it says is true.
From the purpose, derive three to six criteria. Fewer than three and you are probably missing something important. More than six and reviewers cannot hold them all in mind, so their judgements blur together. For a support answer, sensible criteria might be: correctness of the procedure, completeness (does it cover all the steps?), clarity for a non-expert reader, and appropriate tone. For a summary, the criteria might be: coverage of key points, faithfulness (nothing invented), and conciseness. Each criterion should be independent of the others as far as possible. If two criteria always move together, they are really one criterion, and keeping both double-counts it.
Levels need descriptions, not just numbers
A rubric with levels numbered 1 to 5 but no description of what each level means is barely better than no rubric, because reviewers will interpret the numbers differently. Each level needs a short description anchored in observable features of the output. For correctness, level 5 might read: "All steps are correct and in the right order, with no misleading statements." Level 3 might read: "The main steps are correct, but one minor step is wrong or out of order." Level 1 might read: "The procedure is wrong in a way that would cause the user to fail or cause harm." The descriptions should be specific enough that a reviewer can point to the part of the output that justifies the score. If a reviewer cannot explain a score by pointing at the text, the rubric is not doing its job.
Include examples, ideally one strong and one weak output for each criterion, chosen from real outputs of your system. Examples calibrate reviewers far faster than descriptions alone, because they show the boundary between levels rather than describing it. When reviewers disagree, the productive discussion is not about the score. It is about which example the output resembles more closely, and that discussion often reveals that a level description needs tightening. Treat the rubric as a living document: the first version will be imperfect, and the disagreements from the first evaluation round are the material for improving it.
A worked example
Suppose you are evaluating a model that drafts replies to customer emails about delivery problems. You build a rubric with four criteria: factual accuracy (does it state the correct delivery status and policy?), completeness (does it address everything the customer asked?), tone (is it polite and empathetic without being over-familiar?), and actionability (does it tell the customer what happens next and what they need to do?). Each criterion has four levels, from poor to excellent, with descriptions.
You select twenty real customer emails, generate replies with two different prompts, and have two reviewers score all forty replies using the rubric, independently, without knowing which prompt produced which reply. The results show that the two reviewers agree closely on accuracy and completeness, agree moderately on actionability, and disagree often on tone. That pattern is information. It tells you that the tone descriptions need work, perhaps because "over-familiar" means different things to different reviewers, and you tighten the description with examples. It also tells you something about the prompts: suppose prompt B scores higher on completeness but lower on tone, because it packs in more information and reads as abrupt. That trade-off is now visible and discussable, whereas before the rubric, the team would have argued about which prompt was "better" in general, a question with no answer. The rubric does not make the decision for you. It makes the decision discussable, which is the prerequisite for making it well.
A further discipline matters: score blind where possible. Reviewers who know which model or prompt produced an output tend to favour the one they expect to be better, even unconsciously. Hiding that information, and randomising the order of outputs, removes a well-documented source of bias. It costs little and improves the credibility of every conclusion you draw.
Common mistakes
The first mistake is scoring everything on one overall scale, such as "rate this answer out of ten." Overall scores hide trade-offs: an answer can be accurate but rude, complete but confusing, and a single number cannot represent that. Always score by criterion, and report the criterion scores alongside any total. The second mistake is writing level descriptions in vague language: "good," "adequate," "poor." Those words are what the rubric is supposed to define, so using them as definitions is circular. Describe observable features instead. The third mistake is evaluating on cherry-picked examples. Choose your evaluation set before you see the outputs, include difficult and awkward cases, and report results on the full set, not on the examples that flatter your preferred model. The fourth mistake is treating the rubric as finished after one use. Reviewer disagreement is not a failure of the reviewers. It is feedback about the rubric, and the rubric should be revised in response.
Honest limitations
A rubric measures what you chose to measure, and it can miss qualities you did not think to include, such as creativity, cultural sensitivity, or the long-term effect of an answer on a customer's trust. It also cannot fully remove subjectivity; it can only reduce it and make it visible. For high-stakes decisions, such as evaluating a model for medical or legal use, a rubric is a necessary tool but not a sufficient safeguard. Expert review, testing on realistic cases, and careful monitoring after deployment all remain essential. A rubric is also only as fair as its criteria: if the criteria embed a narrow view of what good looks like, the rubric will systematically favour outputs that match that view and penalise legitimate alternatives.
Closing
Define criteria from the application's purpose, describe each level with observable features, calibrate with examples, score blind and by criterion, and revise the rubric when reviewers disagree. To check the prompts you are evaluating, use the Prompt Template Builder, and to catch prompts that may produce unsafe outputs, try the System Prompt Linter. For guidance on writing the prompts themselves, read Write Better AI Prompts.