Introducing Agent Evaluation: An AI Judge That Grades Your Agents

Most teams answer it the slow way — they spot-check a handful of transcripts, trust their gut, and move on. That doesn't scale, and it doesn't catch the agent quietly skipping a step on one conversation in eight. Agent Evaluation is Macha's answer. It's an AI judge that reads back over every conversation an agent actually ran, grades each one against a checklist built from the agent's own instructions, and hands you a percent-followed score plus a one-line reason for every single call.

It's the LLM-as-a-judge pattern — now a standard way to evaluate AI systems — pointed squarely at your support agents and wired into the same dashboard where you build them. This post is a real walkthrough: I ran an evaluation over 18 live conversations from one of our agents and I'll show you exactly what came back, warts and all.

Why "did it deflect?" was never the right question

For years the headline metric for support automation was deflection: did the bot keep the ticket away from a human? But as the 2026 CX writing has caught up, a bot can give a wrong answer in two seconds — that's fast, and it's a failure. Deflection tells you the agent responded. It tells you nothing about whether the agent did the right things to get there.

Agent Evaluation measures the other thing: execution quality. Not "did the customer go away happy" (that's CSAT, and it's noisy), but "did the agent actually search the ticket before answering, follow its escalation rules, stay on-channel, and produce the output it was told to produce." It's the difference between grading the outcome and grading the work. Both matter; only one of them tells you what to fix.

That's also why this is an AI judge and not a dashboard of averages. An LLM judge reviews each conversation against explicit criteria and produces a verdict with a written justification — the same approach Databricks and others now use to score agent accuracy at scale, reaching high agreement with human reviewers while applying the exact same rubric to every conversation (no reviewer drift, no Monday-morning leniency).

Where Agent Evaluation lives

Agent Evaluation is a tab on every agent — it sits right next to Configuration, Chat, Analytics, and History. You don't leave the agent to grade it; you open the agent and ask "how have you been doing?"

Under the hood it runs on the same analysis engine as Studies — Macha's tool for running an AI analysis across thousands of records — so if you've used Studies, the flow will feel familiar. The difference is the source: instead of pointing at raw Zendesk tickets, Agent Evaluation points at Agent Conversations — the actual chats, trigger runs, and sub-agent handoffs this agent has executed.

The builder walks three steps: Scope → Extract → Review. Here's each one with what I actually did.

1. Scope — pick the conversations to grade

First you choose the window. The source is fixed to Agent Conversations, and you set a date range. For this run I scoped it to one week — June 16 to June 23 — which matched 18 conversations. You can grade a day, a sprint, or a quarter; the scope is just how far back the judge reads.

2. Extract — tell the judge what "good" looks like

This is the heart of it, and it's where the feature is cleverest. The prompt reads "What should the AI judge evaluate?" — and rather than make you write a rubric from scratch, Macha will read the agent's own instructions and propose the criteria for you. I clicked build, and it returned "Built 4 fields from the agent's instructions" — turning the things I'd told the agent to do into the things it would now be graded on.

Each field becomes an evaluated column. You pick a field type the same way you do in Studies:

• Single choice — the judge picks exactly one verdict from your list (great for Correctly searched / Partially / Did not / Not applicable).
• Multiple choice — the judge ticks every item that applies (e.g. which pieces of content a summary actually covered).
• Plus the Yes/No, number, and text types for flags, scores, and free-text reasons.

My first field was Ticket Search and Retrieval, a single-choice column with four options and a one-line guidance note — "Judge whether the agent used the ticket tools appropriately…" — so the judge knew exactly how to decide. A Required toggle forces the judge to fill that field on every conversation.

Notice the model selector at the top of this step. It carries an "AI Evaluator" badge, and its subtitle says it plainly: "The model that scores how closely the agent followed its instructions on every conversation it ran." I ran the judge on GPT-5 at 2 credits per conversation. Critically, the judge is a different model doing a different job than the agent it's grading — here the agent runs on GPT-5.4 while the evaluator runs on GPT-5. You're not asking the agent to mark its own homework.

3. Review — see the cost, then run

Before anything executes, you get an Estimate & Run screen: the source, the date range, the fields, the model, the exact count of matching conversations, and the credit estimate.

Mine read 18 matching conversations, 36 credits estimated (GPT-5 at 2 credits each), with my balance shown right below — 2,221.5 credits remaining — and a "Buy a top-up pack" link if a big run would run me short. There's a "Test on 3" button to sample a few conversations first and sanity-check your rubric before committing the whole batch. Like everything in Macha, credits are spent per AI action — here, per conversation the judge reads — and you see the bill before you confirm. I hit Run study.

The verdict: 75% of conversations fully followed instructions

Thirty seconds later, the report. Eighteen conversations processed, 36 credits spent, zero errors, six reportable columns.

The headline is the Instructions Followed donut: 75% followed. Broken down, that's Yes 10 (56%), Partially 7 (39%), and No 1 (6%). In plain terms: this Ticket Summarizer agent did everything it was told on a little over half its conversations, mostly-but-not-entirely on another 39%, and flat-out missed the brief once. That single number is worth more than a month of vibes — and it's a baseline I can now move.

The per-criterion charts are where it gets actionable:

Read that as an operator and the fix-list writes itself. The agent nails channel fit and clarity (94–100%), but there's a single conversation where it didn't retrieve the full ticket before summarizing — and that's almost certainly the same conversation dragging the "instructions followed" number down. One row. The judge found it in a batch I'd otherwise never have re-read.

The grid: a verdict and a reason on every row

The donut is the summary; the results grid is the receipts. Every conversation gets its own row with the judge's verdict, a one-line reason for the call, and the agent's own output side-by-side — so you can click into the "No" and see why the judge marked it down, not just that it did.

This is the part that makes an LLM judge trustworthy: it doesn't just assert a score, it shows its work. A "Partially" with the reason "summarized correctly but did not pull the latest ticket comment" is something you can act on in five minutes. A bare 3.8/5 isn't.

Closing the loop: from grade to fix

A score you can't act on is just anxiety. The nice touch is what sits in the corner after a run: Sidekick offers — "Want me to review this agent's instructions and tools, and suggest improvements? Review my setup." So the path is short: evaluate → see the one criterion that's lagging → ask Sidekick to tighten the instruction that caused it → re-run the evaluation next week and watch the number move.

You can also push a completed evaluation into a Knowledge Source (the "Use as knowledge source" action, top-right of the grid) — turning your graded conversations into searchable context, the same way Studies results can become agent knowledge.

Watch-outs: where an AI judge needs a human

Agent Evaluation is genuinely useful, but it's a judge, not an oracle. A few honest caveats:

• The judge is only as good as your rubric. Build-with-AI gives you a strong first draft from the agent's instructions, but vague criteria produce vague verdicts. Spend the two minutes to add guidance and tight option lists — that's what makes the score mean something.
• LLM judges aren't infallible. Independent research puts good judges at roughly 80%+ agreement with human reviewers, and newer agent-as-a-judge approaches push higher — but that's agreement, not truth. For high-stakes calls, spot-check the judge the way you'd spot-check the agent. Run Test on 3 first and read the reasons; if they don't match your judgment, fix the rubric before the full run.
• It grades execution, not outcomes. "Followed instructions" ≠ "customer was happy." If your instructions are wrong, a 100% score just means the agent is reliably doing the wrong thing. Pair Agent Evaluation with CSAT and resolution data — they answer different questions.
• It costs credits. Each conversation the judge reads spends credits by model (mine was 2 per conversation on GPT-5; the default GPT-5.4 Mini is 1, ranging 0.5–9). The estimate-and-confirm gate exists so a big back-fill is never a surprise — and Test on 3 keeps experimentation cheap.

For most teams none of that is a dealbreaker — it's the difference between knowing your agent skips a step 6% of the time and suspecting it might.

FAQ

What is Agent Evaluation? It's an AI judge built into Macha that grades how closely an agent followed its own instructions across past conversations. You define (or auto-generate) a checklist of criteria, an evaluator model scores every conversation, and you get a percent-followed score, per-conversation verdicts, and a one-line reason for each call.

What is "LLM as a judge"? A pattern where one AI model evaluates another's outputs against explicit criteria, producing a score and a written justification — applying the same rubric to every item with no reviewer drift. Agent Evaluation is that pattern, specialized for support agents and wired into the Macha dashboard.

Does the agent grade itself? No. You choose a separate evaluator model (the one with the "AI Evaluator" badge). In my run the agent ran on GPT-5.4 and the judge ran on GPT-5 — different model, different job.

How much does an evaluation cost? It's credit-based, priced per conversation by the evaluator model you pick, and you see the exact total before you confirm. My run was 18 conversations for 36 credits on GPT-5. Use Test on 3 to validate your rubric cheaply first. See the pricing page for credit details.

Which plans include it? Agent Evaluation is part of Macha's analysis suite (the same family as Studies), currently in Beta, on the Professional and Enterprise plans. Check the pricing page for current plans.

Try it

If you've ever shipped an agent and then wondered, honestly, whether it's doing what you told it to — Agent Evaluation gives you a straight answer in minutes, on conversations that already happened. Start a 7-day free trial, no credit card required, open any agent, and run an evaluation over last week. Or read the Agent Evaluation docs for the full walkthrough, and see how the same engine powers Studies across your whole ticket archive.

Written by Abbas (Customer Support & AI, Macha) · Reviewed by Ankeet Guha (Co-founder & CTO) · Published 2026-06-24 · Last updated 2026-06-24.