Agent Evaluation: Grade How Your AI Agents Actually Performed

Agent Evaluation is Macha's answer. It's an AI judge that runs a checklist over an agent's past conversations — chats, autonomous trigger runs, sub-agent handoffs, embedded chatbot sessions — and scores, for each one, whether the agent followed its instructions, with a one-line reason. You get a headline percentage followed, a per-conversation verdict you can audit in a click, and a checklist that Macha can build automatically from the agent's own prompt. It's quality measurement for agents, with a number you can move.

To write this, I ran a real evaluation over 18 conversations of one of our agents and I'll show you exactly what came back, including where it scored the agent down.

How we wrote this

The screenshots and numbers here are from a real evaluation run on the Macha dashboard — a live agent ("Ticket Summarizer"), its real conversations from the last seven days, judged by GPT-5. Nothing is mocked. Where the agent slipped, the report says so, and we left that in.

What Agent Evaluation is (and isn't)

Agent Evaluation grades past conversations — it's a reviewer, not a live moderator. It doesn't sit in front of your agent stopping bad replies in real time (for that you use the agent's instructions and confirmation gates on write tools). Instead it looks backward over a window of conversations and tells you how the agent actually did, so you can decide whether a prompt change helped, catch a regression, or set a baseline before rolling an agent out wider.

It lives on every agent's detail page, under an Agent Evaluation tab next to Configuration, Chat, Analytics, and History. Because it's scoped to the agent you're already looking at, you can't accidentally evaluate the wrong one.

Setting up an evaluation

Open an agent, click Agent Evaluation, and (the first time) hit Create your first evaluation. You get a three-step wizard: Scope → Extract → Review.

1. Scope — what to evaluate

You set a date range (last 7 days by default), can optionally restrict to conversations that ran on a specific model (handy after a model migration), and — the clever bit — you choose which version of the agent's instructions to judge against:

• The instructions that were live when each conversation ran (recommended) — the honest "did this agent do its job at the time?" review.
• Your current instructions — the "would today's prompt have passed?" regression view.
• A specific saved version — for A/B comparisons between prompt iterations.

That version choice is what makes Evaluation a real before-and-after tool rather than a one-shot score.

2. Extract — let Macha build the checklist

This is the part that saves you an afternoon. Click Build with AI and Macha reads the agent's own instructions, identifies the rules in them, and proposes one evaluation field per rule — name, the right field type, options, and judge guidance — streaming them in as it works.

In my run it built four fields from the agent's instructions in about thirty seconds — things like Ticket Retrieval Workflow Used, Summary Content Coverage, Summary Clarity and Concision, and Delivery Action and Channel Fit. You can edit, delete, or add to anything it proposes, or skip it and define the checklist by hand (field types are Yes/No, single/multiple choice, number, and short/long text, each with guidance for the judge).

The two fields you always get

Every evaluation includes two mandatory fields you can't remove:

• Instructions followed — a Yes / Partially / No verdict. Yes = every required step done correctly; Partially = mostly right but a notable slip (wrong tag, skipped step); No = wrong action or ignored the instructions.
• Why — a one-to-two-sentence reason pointing at a specific moment in the conversation. This is the audit trail: scan the score, click a row, see why — without re-reading the transcript.

3. Review — see the cost, then run

Like Studies, Evaluation makes you look at the bill first. Estimate cost shows the matching conversation count and the credit total (a per-conversation cost set by the judge model), and you confirm before anything runs.

My run: 18 conversations, 36 credits on the GPT-5 judge — two credits per conversation. If your filters return zero conversations, the run button is disabled with a one-click link back to widen the scope.

A real run: what the judge found

The evaluation processed all 18 conversations with 0 errors and produced a report. The headline is a donut:

75% followed. Here's the breakdown, and how that number is computed:

The score isn't just "% Yes." It gives partial credit: Yes + ½ × Partially ÷ total, so 10 + (½ × 7) = 13.5 out of 18 = 75% followed. That's a fairer signal than a pass/fail — an agent that's mostly-right-with-small-slips reads as 75%, not 56%.

The custom fields tell you where it slipped. In this run the agent was strong on retrieval and clarity but not perfect:

• Ticket Retrieval Workflow Used: read the full ticket on 17 of 18 (94%); one conversation had insufficient retrieval.
• Summary Clarity and Concision: 94% clear and concise; one unclear.
• Delivery Action and Channel Fit: 100% used the right channel.
• Digest Grouping Applied: 100% correct.

So the agent's weak spot isn't tone or delivery — it's the occasional shallow retrieval that drags seven conversations from "Yes" to "Partially." That's a concrete, fixable prompt note, and I only know it because the judge graded every conversation, not a sample.

Each verdict is auditable. The results table puts the Instructions followed pill and the Why reason as the first two columns, so you can scan the score and click into any conversation to see the judge's reasoning against the original transcript.

The loop this unlocks

A single score is mildly interesting; the before-and-after is the point.

• Did my prompt change help? Run an evaluation pinned to "instructions as they ran," note the score, edit the prompt, then re-run over the same date range against "latest." Two donuts, one comparison — you see immediately whether you helped, hurt, or did nothing.
• Weekly quality line. Same evaluation, scope = last 7 days, run every week. Watch the percentage trend instead of guessing.
• Regression catch. After any prompt change, re-run on a fixed historical window to confirm you didn't break something the old prompt got right.

You re-run straight from the evaluations list — it reopens the same config so you only change what you want.

A note on scope: when an agent delegates to sub-agents, the judge sees the whole delegation tree — every tool call and sub-agent reply — so a routing agent is graded on whether its handoff was actually correct, not merely on whether it called a tool.

Where it fits with the rest of Macha

Evaluation is the quality counterpart to building and analysing:

• Use Studies to understand your ticket volume and decide what to automate.
• Build the agent, connect its tools, and run it.
• Use Agent Evaluation to confirm it's doing the job — and to keep it honest every time you touch the prompt.

Honest trade-offs: it evaluates the past, not live traffic; the judge is itself an AI model (you pick which), so treat the verdicts as a strong, consistent reviewer rather than ground truth — the Why column exists precisely so you can spot-check the calls; and each run costs a small per-conversation credit amount shown before you commit.

FAQ

What does Agent Evaluation actually score? Whether your agent followed its instructions on each past conversation — a Yes/Partially/No verdict plus a one-line reason — and any custom checks you add (right tool, right tone, right action).

Where do I find it? On any agent's detail page, under the Agent Evaluation tab. It evaluates that specific agent's conversations.

Do I have to write the checklist? No — Build with AI reads the agent's instructions and proposes the checklist for you (you can edit it). Or define fields manually if you have a specific rubric.

How is the "% followed" calculated? (Yes + ½ × Partially) ÷ total conversations. Partial credit for partial work — in our run, 10 Yes + 7 Partially over 18 = 75%.

How much does it cost? A per-conversation credit amount based on the judge model, shown as an estimate before you run. Our 18-conversation run on the GPT-5 judge cost 36 credits. See the pricing page for plans.

Is it real-time? No. It evaluates past conversations. For live guardrails, use the agent's instructions and confirmation gates on write tools.

Try it

If you've shipped an AI agent and only have a gut feel for how well it's doing, an evaluation turns that into a number — and shows you exactly where to tighten the prompt. Start a 7-day free trial, no credit card required, open an agent, and run your first evaluation — or read the docs for the full walkthrough.