How to Measure AI Agent Quality: Evaluation and AI-Judge Scoring

"It feels fine" is not a quality bar. Deflection rate isn't either — an agent can deflect a ticket by giving an answer that's plausible, on-brand, and wrong. To run an AI agent in production with a straight face, you need to measure its quality the way you'd measure a human agent's: against a clear standard, on real conversations, at a scale a person could never review by hand.

This guide is the practical version of that. What "quality" decomposes into for an agent, how LLM-as-a-judge scoring lets one AI grade another's work, where that approach quietly lies to you, and how to actually run an evaluation over your own support history. Macha makes agents that sit on top of your helpdesk — so we care a lot about this question, and we built Agent Evaluation to answer it. I'll show you that feature with real screenshots near the end, but most of this is vendor-neutral: the metrics and the judging method are the same wherever your agent runs.

Why a single number is the wrong place to start

The instinct is to ask for the score — one percentage that says "the agent is 87% good." That number doesn't exist, because an agent does several distinct things and can be excellent at one while failing another.

Modern agent-evaluation practice splits the problem into three levels (Confident AI's agent-evaluation guide lays this out well):

• End-to-end — black-box. Did the conversation reach the user's goal? This is the customer's experience, and the only level that matters to them.
• Trajectory-level — the path the agent took. Which tools did it call, in what order, with what arguments, how many retries? Two agents can both "succeed" while one took three steps and the other took eleven.
• Component-level — the individual pieces. Did the retriever surface the right doc? Did the sub-agent handle the handoff cleanly? This is where you isolate why something failed.

You want a read on all three, because a good end-to-end score sitting on top of a chaotic trajectory is a regression waiting to happen. The agent got lucky; it won't stay lucky.

What you're actually measuring

Underneath those levels sit the metrics. Not all apply to every agent — a simple FAQ responder needs three of them; an autonomous tool-using agent needs all of them. Here's the working set, with a one-line definition each.

The single most useful habit here is knowing which checks need a judge and which don't. Whether the agent called lookup_order with the correct order ID is a string comparison — you don't need a language model to grade it, and you shouldn't, because a deterministic check is faster, free, and never disagrees with itself. Whether the agent's answer was helpful, grounded, and on-task is a judgment call over open-ended text. That's where LLM-as-a-judge earns its place. The rule of thumb across the 2026 literature: deterministic for the exact stuff, judge for the subjective stuff. Mixing the two — a "hybrid eval" — beats either alone.

For a customer-support agent specifically, the metrics that move the needle are usually task completion, faithfulness (did it stay grounded in your help center, or did it freestyle?), answer relevancy, and a safety pass. Tool and argument correctness matter the moment your agent does more than talk — issuing refunds, looking up orders, scheduling.

How LLM-as-a-judge scoring works

LLM-as-a-judge is exactly what it sounds like: you give a language model the conversation, a rubric, and ask it to grade. For each item on your checklist it returns a verdict — pass or fail, or a score on a scale — plus a written reason. Done well, it turns "read 4,000 transcripts" into a job that runs while you get coffee.

It works because the judging task is easier than the original task. Recognizing whether an answer is grounded in a source is simpler than generating the answer in the first place — the same reason it's easier to grade an essay than to write one. The seminal benchmark here, Zheng et al.'s "Judging LLM-as-a-Judge" (the MT-Bench paper), found that a strong judge model agreed with human preferences at a rate comparable to how often two humans agree with each other — north of 80%. That's the bar that made automated evaluation credible at scale.

A few design choices shape how reliable your judge is:

• Binary beats vague scales. "Pass / fail against this specific rule" is far more stable than "rate this 1–10," because a 7-vs-8 distinction is noise. If you need granularity, build it from multiple binary checks rather than one fuzzy number. Frameworks like DeepEval push the same idea: explicit evaluation steps over a one-line criterion.
• Checklists over adjectives. "Was this good?" produces inconsistent grades. "Did the agent (1) confirm the order number, (2) cite a help-center article, (3) avoid promising a refund it can't issue?" produces repeatable ones.
• Single vs pairwise. A single-output judge grades one conversation against the rubric. A pairwise judge compares two candidates — useful when you're A/B testing two agent versions and want to know which is better, not whether each is "good."

The output you want from all this isn't a vibe. It's a percentage-followed score (what share of your checklist the agent actually satisfied across the conversations), a per-conversation verdict, and a one-line reason for each call so a human can audit any grade in seconds.

Where the judge quietly lies to you

This is the section most "AI evaluation" posts skip, and it's the one that saves you. LLM judges are useful, not infallible, and they fail in specific, documented ways:

• Verbosity bias. Judges tend to reward longer, more elaborate answers even when a short one was correct and better. Your most efficient agent can score worse.
• Position bias. In pairwise comparisons, the option presented first (or second) gets a systematic edge regardless of quality. Mitigate by running both orderings and averaging.
• Self-preference / self-enhancement bias. A judge model tends to favor text written in its own style — sometimes its own outputs. Using a different model to judge than the one that generated the answer is a cheap, sensible guardrail.
• Vague criteria = noisy scores. The single biggest source of bad evals isn't the model; it's a rubric so broad the judge interprets it differently each run. Tighten the criteria before you blame the judge.
• No ground truth in production. Live tickets don't arrive with a labeled "correct answer," which limits reference-based metrics. You evaluate on what's checkable — groundedness, policy adherence, relevancy — not against an answer key you don't have.

The non-negotiable: calibrate the judge against humans. Have a person grade a sample of the same conversations, measure agreement (Cohen's kappa or a simple agreement percentage), and only trust the automated score once it tracks human judgment. Then re-check periodically, because models and your product both drift. An LLM judge is a force multiplier on human review — not a replacement for it. And give yourself enough volume: a handful of conversations tells you nothing; 2026 guidance points at the low hundreds of cases before aggregate numbers mean much.

When NOT to lean on an AI judge: for anything with a single objectively-correct value (an order ID, a refund amount, a returned tool name), use a deterministic check. For high-stakes, low-volume decisions — a legal or medical-adjacent reply, a billing dispute over a large sum — keep a human in the loop and use the judge to triage which conversations a human should read first, not to sign off.

Running an evaluation on your own agents

Here's where it gets concrete. Macha's Agent Evaluation grades how your agents actually performed on past conversations — not a synthetic test set, but the real chats, trigger runs, and sub-agent handoffs that already happened on your helpdesk. The flow mirrors Macha Studies: you build a checklist, see the cost before you commit, run it, and read a report.

1. Build the checklist. You define the criteria the AI judge will grade against — the pass/fail rules that encode what "good" means for your agent. You can write them by hand or have Macha draft a starting checklist for you, then edit it down to the rules you actually care about.

2. See the cost before you run. Evaluation is credit-based, like everything else in Macha — credits are spent per AI action, so grading a batch of conversations has a price, and Macha shows you the estimate before anything runs. Nothing is charged until you confirm.

3. Read the per-conversation verdicts. The judge runs your checklist over every selected conversation and returns a result row each: which criteria it followed, which it missed, and a one-line reason for each call. This is the audit trail — you can open any conversation and see exactly why it was graded the way it was.

4. Read the aggregate score. The report rolls every conversation up into a percentage-followed score — the share of your checklist the agent satisfied across the batch — plus a breakdown by criterion so you can see which rule the agent fails most. That's the number you track release over release.

Because it grades agents on top of the helpdesk you already use — Zendesk, Freshdesk, Gorgias, Front — you're measuring the agent against the same conversations your customers actually had, not a lab approximation.

A starter rubric you can copy

If you're staring at a blank checklist, this is a reasonable v1 for a support agent. Keep every item binary, keep them specific, and add the deterministic checks outside the judge.

Start narrow, calibrate against a human-graded sample, then expand. A small rubric you trust beats a sprawling one you don't.

Where evaluation fits next to Studies

Evaluation and Studies are two sides of the same coin. Studies runs an AI analysis across thousands of tickets to tell you what's in your queue — how much is billing, how urgent, what categories. Evaluation runs an AI judge across your agent's conversations to tell you how well it handled them. One profiles the work; the other grades the worker. Together they close the loop: analyze the queue, build the right agents, measure how they do, fix the weak criterion, re-measure.

FAQ

What's the difference between deflection rate and agent quality? Deflection counts how many tickets the agent resolved without a human. Quality asks whether those resolutions were correct. An agent can post a high deflection rate while quietly giving wrong answers — which is exactly why you measure quality separately, on the content of the conversations, not just the outcome counts.

Is LLM-as-a-judge accurate enough to trust? For subjective criteria, a well-designed judge agrees with human raters at rates comparable to human-human agreement (80%+ in benchmark studies). But that depends entirely on a tight rubric and on calibrating the judge against a human-graded sample first. Treat it as a force multiplier on review, not a replacement.

Should I use a scale (1–10) or pass/fail? Pass/fail per specific rule. Fuzzy scales add noise — the difference between a 7 and an 8 is rarely meaningful. If you need a granular score, build it by combining several binary checks.

Can I evaluate against real customer conversations, not a test set? Yes — and you should. Macha's Agent Evaluation grades real past chats, trigger runs, and sub-agent handoffs from your helpdesk, so you're measuring against the conversations your customers actually had. A synthetic test set is useful for regression testing, but production transcripts are the ground truth of how the agent behaves.

How much does running an evaluation cost? It's credit-based — credits are spent per AI action, and Macha shows you the exact estimate (matching conversations × cost) before anything runs. See the pricing page for current plans, or check out other capability walkthroughs on the blog.

Measure it, then improve it

You can't improve what you can't see. The teams running AI agents well aren't the ones with the flashiest demo — they're the ones who decided what "good" means, encoded it as a checklist, and let an AI judge grade every conversation against it so a human only has to read the failures. That's a closed loop, and it's how an agent gets better release over release instead of drifting until a customer catches it.

If you want to grade your own agents, start a 7-day free trial, no credit card required, connect your helpdesk, and run an evaluation over last month's conversations — or read the Agent Evaluation docs for the full walkthrough.

Written by Abbas (Customer Support & AI, Macha) · Reviewed by Ankeet Guha (Co-founder & CTO) · Published 2026-06-25 · Last updated 2026-06-25.