Agent Evaluation

How it works

You set up a checklist once — or let Macha build one from the agent's own instructions — and an AI judge runs it over every past conversation in scope: chat sessions, autonomous trigger runs, sub-agent handoffs, embedded chatbot conversations. Two fields are always present: Instructions followed (Yes / Partially / No) and a short Why. You add your own checks, like "right tool called" or "refund granted only when policy allowed."

The headline score gives partial credit — Yes count plus half the Partially count, over the total — and results come back as a table, a report with charts, and a per-conversation reason so you can audit a verdict without re-reading the whole thread.

Auto-Eval vs manual evaluations

Every workspace gets Auto-Eval free on every plan: from the moment you create an agent, Macha grades every autonomous run against its live instructions on a hardcoded judge, with no conversation cap and at no cost. Manual evaluations, also included on every plan, are the configurable flow — pick the scope, define a custom rubric, choose the judge model, and evaluate against a specific version of the instructions.

This is how you measure a prompt change: run before and after and compare the scores, instead of guessing whether the edit helped. It evaluates past conversations, not in-flight ones — for real-time control, use the agent's instructions and confirmation gates.