What to Measure: Building an Evaluation Checklist for Support Agents

The instinct is to watch a top-line number: deflection rate, CSAT, "tickets touched." Those tell you something, but they don't tell you why a conversation went wrong, and they hide the failures you most need to catch — the agent that promised a refund it wasn't allowed to give, the one that invented a policy, the one that should have escalated and didn't. AI support agent evaluation is the discipline of grading individual conversations against an explicit checklist, so you can see exactly where behaviour drifts and fix the criteria instead of guessing.

This is a practical guide to building that checklist: which dimensions actually matter for a support agent, how to turn fuzzy expectations into criteria a judge can score, and how to run it across your real conversation history. We'll use Macha's Agent Evaluation feature for the worked examples — Macha is the AI agent layer that sits on top of your helpdesk (Zendesk, Freshdesk, Gorgias, Front), so the agents being evaluated are the ones already working your live queue.

Why a checklist beats a single score

A single satisfaction or deflection number is a lagging signal. By the time CSAT dips, the bad behaviour already shipped to hundreds of customers, and the metric won't tell you which of a dozen possible causes is to blame. Worse, the headline numbers are easy to game: an agent that punts every hard question to a human can post a great "no escalation needed" score on the easy ones while quietly failing the cases that matter.

Two findings from the evaluation literature are worth internalising before you measure anything:

• Deflection is not resolution. Fin AI's customer-service evaluation framework is blunt about this — counting a redirect, a deflection, or a closed chat as a "resolution" inflates your numbers without solving anyone's problem. The honest version is resolution durability: did the issue stay resolved for 7–10 days without the customer coming back? (Ada, Fin AI)
• Easy-only test sets lie. If you grade your agent only on the simple tickets, projected performance can be overstated by 20–30 percentage points versus a representative mix that includes edge cases and emotionally charged conversations.

A checklist fixes both. Each conversation is scored against the same explicit criteria, so a high score has to be earned on hard tickets, not handed out for dodging them. And because every criterion is separate, the result is diagnostic: you don't just learn "the agent is 78% good," you learn which rule it breaks and on which conversations.

The dimensions worth measuring

Across the better evaluation frameworks — Fin AI's six dimensions, Maxim's production checklist, Confident AI's metric taxonomy — the same families of criteria show up. Here's the set that actually matters for a support agent, with the question each one answers.

A few notes on using this table well:

• Split deterministic from subjective. Some of these can be checked programmatically (did the agent call the right tool? did it stay under N turns?). The rest — tone, helpfulness, "did it actually resolve" — are judgement calls. Maxim's guidance is the right rule of thumb: reserve an LLM judge for the subjective qualities and keep hard checks for the objective ones.
• Weight by risk, not by frequency. A tone miss is a bad day; a wrong refund or a hallucinated policy is a chargeback or a compliance incident. Treat the safety and policy criteria as gates, not averages.
• Don't measure everything at once. Five to eight sharp criteria you actually act on beat thirty you never read.

Turning a dimension into a scoreable criterion

A checklist is only as good as the criteria are specific. "Was the response good?" is unscoreable — two reviewers (human or AI) will disagree. The fix is to write each criterion as a single, testable statement with explicit pass/fail guidance.

Compare:

❌ "The agent handled the refund well." ✅ "If the customer requested a refund, the agent (a) checked the order against the refund-eligibility policy, (b) only issued a refund within the allowed window, and (c) clearly stated the outcome. Mark FAIL if a refund was issued outside policy or the outcome was left ambiguous."

The second version tells a judge exactly what to look for and when to fail. Three habits make criteria robust:

1. One behaviour per criterion. If you find an "and" doing heavy lifting, split it — otherwise a partial pass becomes a coin flip.
2. Name the failure explicitly. Stating the FAIL condition is what makes scores consistent across thousands of conversations.
3. Make it binary or banded. A pass/fail or a small fixed scale aggregates cleanly; a vague 1–10 invites drift.

In Macha you can draft these criteria by hand or describe the behaviour you want and let the builder turn it into structured checklist items — the same build-with-AI pattern used elsewhere in the product.

Build a representative sample — not a happy-path one

Your checklist runs against a set of conversations, and that set is where most evaluations quietly go wrong. If you only grade the tickets the agent obviously got right, you'll measure your own optimism.

A defensible sample includes, in roughly the proportion they occur in your real queue:

• Bread-and-butter tickets — the WISMO ("where is my order"), password resets, and FAQ lookups that make up the bulk of volume.
• Multi-step workflows — anything that needs a tool call, a lookup across systems, or conditional logic.
• Edge cases and failure modes — missing order numbers, ambiguous requests, tool timeouts.
• Emotionally charged conversations — angry, confused, or at-risk customers, where tone and escalation matter most.

Because Macha agents run on top of your helpdesk, the sample is just your own history. Agent Evaluation runs over real past conversations — every chat, every autonomous trigger run, and every sub-agent handoff — so you're grading the agent on exactly the traffic it handled, not a synthetic test set. Pull a window of recent tickets, or scope to a specific agent, and you have a representative batch in a click.

Running the evaluation: an AI judge over your history

Here's where the checklist becomes a score. Macha's Agent Evaluation works like a judge: you point it at a batch of conversations, attach your checklist, pick a model, and it reads every conversation and marks each criterion pass or fail — with a one-line reason for each call, so you can see why it scored the way it did.

Like every AI action in Macha, the judge costs credits, and like Studies, it makes you look at the bill first. Before anything runs you get an estimate — the exact number of conversations matched, the credit cost per conversation for the model you chose, and the total — and nothing is charged until you confirm.

On cost: credits are charged per AI action, not per "good score." Evaluation is one judge action per conversation, billed on your credit balance. The default GPT-5.4 Mini model is 1 credit; heavier reasoning models cost more (GPT-5 is 3, GPT-5.4 is 5). For a first pass, a mini model over a few hundred conversations is cheap; reserve the expensive models for criteria where nuance genuinely changes the verdict. See the pricing page for current rates.

When it runs, you get a results grid — one row per conversation, one column per criterion — that you can sort and filter to find exactly the failures you care about.

And the payoff is the report: an overall percentage-followed score across the batch, broken down by criterion, so you can see at a glance which rules the agent honours and which it breaks. Click into any criterion to read the conversations behind the number.

This is the loop that makes the checklist worth building. The score tells you how often the agent follows each rule; the per-conversation reasons tell you why it doesn't; and you fix the agent's instructions, knowledge, or tools — then re-run the same checklist and watch the number move. It's the analysis counterpart to the live agent, the same way Studies is for ticket content.

Calibrate the judge — it isn't ground truth

An LLM judge is fast and consistent, but it is not an oracle, and treating its score as gospel is the most common way these programs go wrong. Two habits keep it honest:

• Spot-check against humans. Have a person grade a small sample — 20–50 conversations — and compare to the judge's verdicts. Where they disagree, the criterion is usually too vague, not the judge being dumb; tighten the wording and re-run. Maxim's framework is explicit that judge scores should be calibrated against human-rater samples before you trust them at scale.
• **Re-read the reasons, not just the score.** Macha gives a one-line reason per call precisely so you can audit it. If the reasons are sound but you disagree with the verdict, your rubric is the bug.

Watch-outs and when not to bother

Evaluation is a force multiplier, but it isn't free and it isn't always the right move. Be honest about the limits:

• It costs credits. Running a judge over thousands of conversations is real spend — the estimate-and-confirm gate exists so it's never a surprise, but budget for it and start with a sample.
• A judge can be wrong in correlated ways. If your criterion is ambiguous, the judge will be consistently wrong, which looks like signal. Calibration is non-negotiable.
• Don't evaluate what you won't act on. A dashboard nobody reads is worse than no dashboard. If you can't change the agent's instructions, knowledge, or tools in response, you don't need the metric yet.
• Tiny volumes don't need this. If you're handling a handful of conversations a day, read them. Formal evaluation earns its keep when volume makes manual review impossible — which is exactly when the failures hide.
• The checklist drifts. As your product and policies change, criteria go stale. Revisit the checklist on the same cadence you revisit your macros and help-centre articles.

A starter checklist you can lift

If you want a v1 to adapt, this set covers the high-risk behaviour for most support agents without ballooning into noise:

1. Resolved — the agent fully addressed the request with no open question left dangling.
2. Grounded — every factual claim is supported by retrieved knowledge or tool output; no invented policies, prices, or dates.
3. Policy-correct — refunds, cancellations, and account actions followed documented policy and used the correct tool.
4. Escalated correctly — out-of-scope or high-risk cases were handed to a human with a clear summary.
5. On-tone — concise, on-brand, and appropriately empathetic to the customer's mood.
6. In-bounds — no promises outside policy, no exposure of another customer's data.
7. Efficient — reached resolution without redundant tool calls or needless back-and-forth.

Run that across a representative batch, calibrate against a human spot-check, fix the two worst-scoring criteria, and re-run. That single loop will tell you more about your agent than any deflection chart.

FAQ

What's the difference between AI support agent evaluation and CSAT? CSAT measures what the customer felt after the fact; evaluation measures whether the agent followed your rules on each conversation, with a reason for every call. CSAT tells you something is wrong; a checklist tells you what and where. Use both.

What should an AI support agent evaluation checklist include? At minimum: resolution/correctness, grounding (anti-hallucination), policy and action correctness, escalation quality, tone, safety/boundaries, and efficiency. Write each as a single testable pass/fail statement with an explicit failure condition.

Can an AI reliably grade another AI? For subjective qualities — tone, helpfulness, "did it resolve" — an LLM judge is reliable once calibrated against a human-graded sample. For objective checks (right tool, turn count), prefer deterministic rules. Always re-read the judge's reasons, not just its score.

How much does it cost to run an evaluation in Macha? It's credit-based — one judge action per conversation, priced by the model you pick, with the exact total shown before you confirm. A mini model over a sample is inexpensive; see the pricing page for current rates.

Does Macha evaluate live agents or just test cases? Real ones. Because Macha agents run on top of your helpdesk, evaluation runs over your actual conversation history — chats, autonomous trigger runs, and sub-agent handoffs — not a synthetic test set.

Start measuring

A good evaluation checklist turns "I think the agent is doing fine" into a number you can defend and a list of fixes you can act on. Write seven sharp criteria, run them over a representative batch, calibrate against a human read, and iterate. Start a 7-day free trial, no credit card required, connect your helpdesk, and run your first evaluation — or read the Agent Evaluation docs for the full walkthrough. If you're standing this up on Zendesk specifically, the Macha on Zendesk page shows how the agents being graded plug into your existing queue.

Written by Abbas (Customer Support & AI, Macha) · Reviewed by Ankeet Guha (Co-founder & CTO) · Published 2026-06-24 · Last updated 2026-06-24.