Agent Evaluation

How it works

You set up a checklist once — or let Macha build one from the agent's own instructions — and an AI judge runs it over every past conversation in scope: chat sessions, autonomous trigger runs, sub-agent handoffs, embedded chatbot conversations. Two fields are always present: Instructions followed (Yes / Partially / No) and a short Why. You add your own checks, like "right tool called" or "refund granted only when policy allowed."

The headline score gives partial credit — Yes count plus half the Partially count, over the total — and results come back as a table, a report with charts, and a per-conversation reason so you can audit a verdict without re-reading the whole thread.

Auto-Eval vs manual evaluations

Every workspace gets Auto-Eval free on every plan: from the moment you create an agent, Macha grades its conversations against its live instructions on a hardcoded judge, up to 50 conversations per agent, at no credit cost. Manual evaluations (Professional and Enterprise) are the configurable flow — pick the scope, define a custom rubric, choose the judge model, and evaluate against a specific version of the instructions — billed one credit per evaluated conversation at the judge model's rate.

This is how you measure a prompt change: run before and after and compare the scores, instead of guessing whether the edit helped. It evaluates past conversations, not in-flight ones — for real-time control, use the agent's instructions and confirmation gates.