Agent Simulations: Dry-Run Your Agents on Real Records

What Agent Simulations are

Agent Simulations let you run an agent against a pool of real records (today, past Zendesk tickets) with write tools stubbed, and see how the agent behaves before it ever touches a live customer. Instead of shipping a prompt change and hoping, you replay it against last week's tickets, watch the conversations that would have happened, and read the AI judge's verdict on each one.

Simulations live under the Simulations tab on every agent's detail page, next to Configuration, Try it, Evaluation, and History.

How Simulations differ from Evaluation and Studies

Three related but distinct features. Easy to conflate; worth being precise about:

• Agent Evaluation grades your agent's past real conversations after the fact. The conversation actually happened, real tools ran, real customers were on the other end. Evaluation tells you whether the agent did the right thing on runs that already shipped.
• Studies run an AI extractor across a list of records and fill in a spreadsheet-shaped grid: no agent involved, no tool calls, just structured extraction.
• Agent Simulations take a fresh set of records and run the agent through them with tools stubbed. New conversations are generated; the agent decides what to do; write tools return canned responses instead of firing for real. It's the closest thing to a dry run of production behaviour.

Use Evaluation to audit what happened. Use Studies to extract structured facts from records. Use Simulations to preview what would happen if you shipped this version of the agent.

When to reach for a Simulation

Three situations where Simulations pay off:

• You've changed the agent's prompt or added a sub-agent and want to see the new behaviour on realistic tickets before turning the trigger back on.
• You're rolling out an agent to a new customer and need to prove it handles their real ticket mix (not synthetic examples) without touching their live queue.
• You want to sanity-check a tool wiring. Simulations show every tool call the agent would have made, with the stubbed response it received, so you can catch missing tools, wrong arguments, or accidental confirmations before production traffic sees them.

How a Simulation is shaped

Every Simulation has four parts:

• The agent: preselected from the page you're on; you can't accidentally run the wrong one.
• An adapter: where the records come from. Today: Zendesk Tickets. More adapters land as new sources come online.
• Adapter config: the scope of records the run will process. For Zendesk that's a search query plus an optional date range, exactly like the Studies scope step.
• A model snapshot: the run freezes whichever model the agent is on at launch time, so every conversation this run spawns pays for that model, not whatever the agent has now. Later prompt or model edits don't invalidate the results.

Creating and running a Simulation

Open any agent and click the Simulations tab. The first run lands on a zero state explaining the feature with a primary Create your first simulation button. After that it shows a table of past simulations with their status, record count, and verdict counts.

Clicking New simulation walks you through a three-step wizard:

1. Source: "Which records should we replay?" Pick the adapter (Zendesk Tickets today), the connector instance, and the query + date range that defines the record pool.
2. Method: "How should the agent be tested?" Configure any optional skip filters (e.g. drop auto-replies before spending credits on them) and decide how many records to cap the run at. Small test runs (a handful of tickets) let you preview simulation behaviour without burning your whole scope.
3. Estimate & Run: review the config summary, see the credit estimate before launching, and hit Run. If the scope returns zero records, the button is disabled with a one-click link back to widen the scope.

What happens during a run

The run engine walks the record pool one ticket at a time, spawning a fresh conversation per record. For each one:

1. The agent reads the ticket, plans its response, and starts calling tools.
2. Read tools (searching knowledge, reading Zendesk, fetching Shopify orders) fire for real against the customer's data, because the simulation needs real context to be realistic.
3. Write tools (adding replies, updating statuses, tagging tickets) are stubbed by the tool executor. They return a plausible canned response so the agent believes the write succeeded and continues its reasoning, but no side effect lands on the actual ticket.
4. The full transcript is saved as a Conversation doc with source: "agent-simulation" so it never mixes with real chat history.
5. Auto-Eval runs on the conversation and lands a StudyResult verdict (Instructions followed, Why, Resolution, Sentiment, Category) using the same grader used elsewhere.

Progress streams live: a "0 of N processed" counter, rows appearing in real time, and per-record verdicts filling in as the judge scores them.

The results view

Every simulation lands on its own results page with three things:

• A summary strip: adherence percentage, verdict breakdown (Yes / Partially / No), record count processed, and total credits spent.
• A per-record table: one row per ticket with the auto-eval verdict, resolution, sentiment, and the judge's reasoning. Click any row to open the full generated conversation, including every stubbed tool call and the response the agent received.
• A conversation viewer: the reasoning trace, tool calls, and final reply the agent produced for that ticket, exactly as they would have rendered in the dashboard chat. Stubbed writes are marked so you can see what would have shipped had the run been live.

Everything the simulation produces is filtered out of the normal agent-evaluation view. Simulated conversations don't count towards your auto-eval percentages, and they don't show up in Past conversations for other users on the org.

Snapshots and re-running

Every simulation freezes its config at launch: the adapter key, the adapter config, and the model the agent was on. If you later change the agent's prompt, swap its model, or delete the simulation config, past runs stay valid. The snapshot on AgentSimulationRun means the results are always readable in the context they were produced.

From the simulations list you can Run again on any past simulation. That reopens the wizard with the original config loaded, which is useful for weekly regression runs against the same historical window after every prompt change, or for A/B comparing two prompt versions against the same ticket pool.

What Simulations don't do

• They don't hit customers. Write tools are stubbed. If your agent's job includes sending replies via Zendesk, adding tags, or updating statuses, none of that lands on the real ticket during a simulation.
• They aren't live moderators. A simulation is a batch replay, not a runtime guardrail. For real-time controls, use the agent's own instructions or a confirmation gate on write tools.
• They don't guarantee production parity. Read tools fetch live data at simulation time, not at the ticket's original creation time. If a Shopify order has since been refunded, the agent will see the refund even though the customer wrote in before it happened. Simulations are a strong preview signal, not a time-machine.

Plan availability

Agent Simulations are available on the Professional and Enterprise plans. Each simulated conversation costs one agent-run at the model's per-conversation rate, plus the auto-eval judge credit: you see the estimate before launching the run. Read-tool API calls to your connectors also count against those connectors' own rate limits, so keep the record cap sensible on large ticket pools.