Shopify
Stripe
Slack
Notion
Google Workspace
Confluence
How to Run a 30-Day AI Support Agent Pilot (2026): Week-by-Week Plan and Go/No-Go Scorecard
A 30-day AI support agent pilot tests one ticket category, on one help desk, against numbers you wrote down before day 1. Pick the category with a scoring table (volume, risk if wrong, data the agent can reach), capture a baseline, run the agent in safe mode with drafts as internal notes for about two weeks, move a slice of the category to public replies in week 3, read at least 200 tickets, and decide with a go/no-go scorecard on day 30. The pilot answers one question: should this agent handle this category for real?
Key takeaways
- A 30-day AI support agent pilot should cover one ticket category, run in safe mode with internal-note drafts for about two weeks, then put a slice of public replies live in week 3.
- Reading 200 sampled tickets measures a 90% accuracy rate to within about 4 points at 95% confidence; 50 tickets leaves a margin of about 8 points.
- Zendesk bills only verified resolutions, Intercom deducts a Fin resolution when the customer returns, and Gorgias reclassifies an AI resolution if an agent replies within 72 hours.
- Free trials run shorter than a pilot: Gorgias 7 days, Intercom and Zendesk 14 days, Front Autopilot 30 days with up to 1,000 resolutions, and Fini's Enterprise pilot 90 days.
- One action outside the written limits that reaches a customer should end the pilot as a no-go, however good the accuracy and CSAT numbers look.
Disclosure: Macha publishes this blog and sells an AI agent with done-for-you setup, so our team runs pilots like this one with customers. The plan works on any route: your own admin on the help desk's native AI, a partner, or a vendor's team. We checked every outside source on October 14, 2026, and the thresholds below are our recommendations, not an industry standard.
What does a 30-day pilot look like at a glance?
| Days | Stage | What happens | Customer sees AI text? |
|---|---|---|---|
| -7 to 0 | Prepare | Pick the category, record the baseline, sign the charter | No |
| 1 to 5 | Build and test | Knowledge, tools and instructions; test on 50 to 100 past tickets | No |
| 6 to 14 | Safe mode | Drafts land as internal notes on live tickets; people send or fix them | Only after a person sends it |
| 15 to 23 | Public slice | One channel, brand or tagged share of the category gets public replies | Yes, for the slice |
| 24 to 30 | Full category and decision | Whole category live if the slice held; scorecard on day 30 | Yes |
This page is the pilot itself. If you're still choosing vendors, the RFP template has a pilot clause to send to bidders. For the full build, the implementation checklist lists 76 pre-launch items. And if a pilot has already gone wrong, why AI support pilots get switched off is the post-mortem.
How do you pick the one ticket category to pilot?
Score each candidate category on five criteria, 1 to 5, and weight them. Volume and risk carry the most weight because a pilot needs enough tickets to measure and a low cost when it's wrong.
| Criterion | Weight | 5 means |
|---|---|---|
| Volume share | x3 | One of your top two categories |
| Risk if wrong (inverted) | x3 | A wrong answer costs an apology, not money |
| Data the agent can reach | x2 | Answer is in the help center or one lookup away |
| Policy is written and stable | x1 | Written, unchanged for months |
| Outcome is measurable | x1 | Clear done state (customer has tracking) |
Here's the table filled in for a synthetic ecommerce queue. The maximum is 50.
| Category | Vol | Risk | Data | Policy | Measurable | Score |
|---|---|---|---|---|---|---|
| Order status | 5 | 4 | 4 | 5 | 5 | 15+12+8+5+5 = 45 |
| Return policy questions | 3 | 4 | 5 | 4 | 4 | 9+12+10+4+4 = 39 |
| Product questions | 3 | 4 | 3 | 3 | 2 | 9+12+6+3+2 = 32 |
| Refund requests | 4 | 1 | 3 | 3 | 4 | 12+3+6+3+4 = 28 |
| Address changes | 2 | 2 | 3 | 4 | 5 | 6+6+6+4+5 = 27 |
| Billing disputes | 1 | 1 | 2 | 2 | 2 | 3+3+4+2+2 = 14 |
Order status wins: high volume, one lookup, low cost when wrong. Refund requests score high on volume and lose on risk, which makes them a good second category. In B2B software queues the winner is usually "how do I" questions answered from the docs.
Two rules override the score. Pick a category with at least a few hundred tickets in 30 days, or you won't have enough to read. And don't pick anything on your always-human list (chargebacks, legal threats, safety), however well it scores.
What baseline should you capture before day 1?
Without a baseline, day 30 turns into an argument about whether the numbers are good. Pull the last 30 days for the pilot category only, not the whole queue.
| Baseline metric | Where it comes from |
|---|---|
| Monthly volume | Help desk report filtered by tag or field |
| First reply and resolution time | Help desk report |
| Handle time per ticket | Time tracking, or 20 timed tickets |
| Reopen and repeat-contact rate within 7 days | Search on reopened tickets and same requester |
| CSAT | Your CSAT survey, same period |
| Human cost per ticket | Loaded hourly cost x handle time |
| Current accuracy | Read 30 human-answered tickets against policy |
The last row is the one teams skip. If your own team gets 94% of a category right against written policy, a 97% accuracy bar for the agent is a moving goalpost.
What scope and guardrails should the pilot run under?
The stages in the table above are three levels of exposure, and each has to earn the next. In safe mode, people send, edit or discard every draft. The public slice is part of the category (one channel, email before chat, or one brand, or a routing rule in your help desk) while the rest stays in safe mode, which gives you a side-by-side comparison.
Write these guardrails into the charter before day 1:
- Always-human topics, with the words that trigger a handoff.
- Action limits with amounts. Read-only is the safest first pilot.
- Handoff format: right group, a summary note, the customer told a person is coming (handoff patterns).
- Rollback rule: our suggestion is one wrong action or two wrong policy answers in a week sends the category back one stage, not the whole agent.
What does the 30-day plan look like week by week?
Days -7 to 0. Score the categories, record the baseline, name the go/no-go owner and two reviewers, sign the charter at the end of this page, and grant access early; security reviews run on their own calendar.
Days 1 to 5. Build from how your best agent answers today, then test on 50 to 100 past tickets: normal cases, escalations, edge cases and tone (testing on real data covers the mix). Don't start safe mode until those results would pass the scorecard.
Days 6 to 14. Reviewers handle every draft the same day and tag it sent unchanged, light edit, heavy edit or discarded, with a reason for the last two. Fix misses by cause: missing knowledge, wrong instruction or wrong tool result.
Days 15 to 23. Switch the slice only after a full safe-mode week at the bar. Reviewers sample 10 to 15 public replies a day, plus every reopen and repeat contact.
Days 24 to 30. Widen to the whole category if the slice held, pull the numbers on day 28 and decide on day 30. Compare the tallies at days 10, 20 and 30: a rate that is still climbing at day 30 argues for an extension, a flat one for a decision.
The daily check takes 10 to 15 minutes: discarded and heavily edited drafts by cause, handoffs (right group, summary, a person replied), reopens on AI replies, every action against its limit, and the running tally.
What should you measure, and how does each vendor define resolution?
Vendors report resolution in their own units, and those units decide what they bill. Read the definition before you put a vendor's dashboard number in your scorecard.
| Vendor | What counts as resolved | What undoes it | Source |
|---|---|---|---|
| Zendesk AI agents | Verified resolution: an LLM confirms the AI resolved it; only this tier bills. Contained resolution and assisted escalation don't bill | Email conversations end 72 hours after the first email; messaging 2 hours by default, up to 72 | Zendesk resolution tiers |
| Intercom Fin | Confirmed (the customer says it helped) or assumed (the customer leaves without asking for more) | The customer returns to the same conversation, even in a later billing period | Fin outcomes |
| Gorgias AI Agent | Automated interaction: a request fully resolved without a human, by AI Agent or another automation | An agent replies within 72 hours, and it bills as a helpdesk ticket instead | Gorgias pricing FAQ |
| Front Autopilot | Billed per conversation as triage ($0.05), handoff ($0.39) or resolution ($0.89) | Not stated in the article | Front Autopilot |
The incentive is plain: a vendor billing per resolution earns more when its definition of resolved is generous, and "the customer stopped replying" is the generous end. That doesn't make the numbers dishonest. It means your scorecard needs a resolution measure the vendor doesn't control. Ours is durable resolution: no reopen, no repeat contact on the same issue within 7 days, and no person had to finish the job. More in what deflection rate measures and Zendesk automated resolutions. Note that Fin's $0.99 is per outcome, and outcomes include procedure handoffs, not only resolutions.
Measure these for the pilot category only: accuracy on sampled tickets (correct and policy-compliant / read), draft acceptance in safe mode (sent unchanged or lightly edited / all drafts), durable resolution, escalation quality (handoffs with the right group and a usable summary / handoffs read), CSAT on AI-touched tickets, handle time saved, and cost per resolved ticket: (AI bill + reviewer hours x loaded rate) / durably resolved tickets.
A worked cost example with synthetic numbers: a $300 AI bill plus 20 reviewer hours at $30 an hour is $900. Divide by 300 durably resolved tickets and the pilot cost $3.00 per resolved ticket. If your people cost $30 an hour at 6 minutes a ticket, that's also $3.00, so this pilot breaks even in month one. The implementation cost breakdown has the budget side.
How many tickets should you read during the pilot?
More than most teams plan for. The margin of error on an accuracy rate depends on how many tickets you read, not on how many the agent handled. At 95% confidence, for an accuracy around 90%, the margin is 1.96 x the square root of (0.9 x 0.1 / n):
| Tickets read | Margin on a 90% accuracy rate | What it supports |
|---|---|---|
| 50 | About 8 points | A rough read after the first days of safe mode |
| 100 | About 6 points | A weekly check |
| 200 | About 4 points | The go/no-go decision |
| 400 | About 3 points | A close call, or a high-risk category |
So "92% accurate on 50 tickets" could be 84% or 100%. Read at least 200 sampled tickets across the pilot for the decision, chosen at random from the category, plus every ticket that reopened, escalated or involved an action.
For rare failures, use the rule of three. If you read n tickets and find zero wrong actions, the true rate could still be as high as about 3/n at 95% confidence. Zero in 100 means up to about 3%; zero in 300 means up to about 1%. That's why action-taking categories need bigger samples than question-answering ones. To grade consistently across reviewers, write the criteria down first; our evaluation checklist has a five-part rubric.
What does the go/no-go scorecard look like?
These thresholds are our recommendations. Agree your own numbers in the charter before day 1 and don't change them after you've seen the data.
| Criterion | Go | Extend 2 weeks | No-go |
|---|---|---|---|
| 1. Accuracy on at least 200 sampled tickets | 90% or higher | 80% to 89% | Under 80% |
| 2. Actions outside written limits that reached a customer | None | None | One or more (knockout) |
| 3. Draft acceptance in the last safe-mode week | 9 in 10 or better | 7 to 8.9 in 10 | Under 7 in 10 |
| 4. Escalation quality on handoffs read | 95% or higher | 85% to 94% | Under 85% |
| 5. Reopen and repeat-contact rate vs human baseline | No more than 2 points higher | 2 to 5 points higher | More than 5 points higher |
| 6. CSAT on AI-touched tickets vs baseline | Within 3 points | 3 to 10 points lower | More than 10 points lower |
| 7. Cost per resolved ticket vs human cost | Lower | Within 20% higher | More than 20% higher |
| 8. Reviewer time in week 4 | 15 minutes a day or less | 15 to 30 minutes | Over 30 minutes |
The decision rule: go if every row is Go, or up to two rows are Extend and neither is row 1 or 2. No-go if row 1 or row 2 is No-go, or three or more rows are No-go. Anything else is extend, once, for two weeks with the same scorecard. A pilot that needs a second extension has given you its answer.
A no-go on this category isn't a no-go on AI. Often the category was the wrong first pick, which is what the scoring table is meant to prevent.
What do vendors' trials and pilots actually give you?
Most trials end before a 30-day pilot does.
| Vendor | What's offered (checked October 14, 2026) | Fits a 30-day pilot? |
|---|---|---|
| Zendesk | 14-day trial of Suite Professional; AI feature access "may vary" during the trial (pricing) | Build and part of safe mode |
| Intercom | 14-day free trial that "includes all features, Fin, and add-ons"; Fin is $0.99 per outcome afterwards (pricing) | Build and safe mode; pay for weeks 3 and 4 |
| Gorgias | 7-day trial with all features, no payment details needed (pricing) | The build only |
| Front Autopilot | 30-day trial with up to 1,000 resolutions; rules stop running at the end unless you opt in (Front) | Yes, if you watch the cap |
| Fini | 90-day free pilot on the Enterprise plan, "live in 30 days" (pricing) | Yes, for Enterprise buyers |
| Macha | Trial with $50 of free usage (about 125 tickets), no credit card, no time limit, including the same done-for-you setup as a paid plan (pricing) | Build and safe mode on about 125 tickets; the public weeks on a paid plan |
Fini's pricing page also promises that if it doesn't reach 90% resolution in 90 days, you pay nothing, for eligible new Enterprise customers who send it 1,000 real tickets. That's measured on Fini's own resolution definition, so score it with your durable number too. For Fin's own stages see our Fin implementation plan; for Autopilot billing after the trial, the Front cost page.
Which mistakes sink a pilot?
Four show up most: piloting three categories at once, so none gets enough reads; writing the success criteria after seeing the data; reporting the vendor's resolution rate without a durable number next to it; and a trial clock that ends at day 14 with no paid weeks lined up. Two smaller ones: nobody owns the daily review, so drafts go stale, and fixes made mid-sample without a dated log, so weeks 2 and 3 measure different agents. The rollback post covers what happens when these go unfixed. The 12 implementation mistakes post covers the planning mistakes outside the pilot itself, such as the security review, billing units and peak-season timing.
Copy the pilot charter
Paste this into a doc and fill in the brackets before day 1.
AI SUPPORT AGENT PILOT CHARTER (template: getmacha.com, Oct 2026)
Category: [name] Channel(s): [email / chat] Help desk: [name]
Dates: prepare [ ] build [ ] safe mode [ ] public slice [ ] decision [day 30]
Go/no-go owner: [name] Reviewers: [name], [name] Implementer: [name/vendor]
BASELINE (last 30 days, this category only)
Volume [ ] First reply [ ] Resolution time [ ] Handle time [ ]
Reopen/repeat contact (7 days) [ ]% CSAT [ ] Human cost per ticket $[ ]
Human accuracy on 30 read tickets [ ]%
SCOPE AND GUARDRAILS
In scope: [question types] Out of scope: [question types]
Always human: [chargebacks, legal threats, safety, VIP list, ...]
Actions allowed: [none / list with limits and amounts]
Handoff: [group], summary note, customer told a person will reply
Rollback: [1 wrong action or 2 wrong policy answers in a week] -> back one stage
STAGES
Safe mode: drafts as internal notes; tag sent / light edit / heavy edit / discarded
Public slice: [channel / brand / routing rule], only after 1 week at the bar
Full category: only if the slice holds the scorecard
MEASUREMENT
Sample: at least 200 random tickets read + every reopen, escalation and action
Durable resolution = no reopen or repeat contact in 7 days, no person finished it
Changes to the agent logged with date and reason
GO / NO-GO (agreed before day 1)
Accuracy >= [90]% Actions outside limits = 0 Draft acceptance >= [9 in 10]
Escalation quality >= [95]% Reopen within [2] pts of baseline
CSAT within [3] pts Cost per resolved ticket < human Review <= [15] min/day
Decision rule: [as agreed] One 2-week extension maximum
Signed: [go/no-go owner] [implementer] Date: [ ]Who does what during the pilot?
| Job | DIY on native AI | Partner or freelancer | Vendor-led (Macha) |
|---|---|---|---|
| Pick the category | You | Partner in discovery | Macha's team, from an analysis of your past tickets; you approve |
| Build and test on past tickets | You | Partner | Macha's team |
| Review drafts in safe mode | You | You | You, 10 to 15 minutes a day; Macha's team fixes misses |
| Fix misses after go-live | You | You, or a retainer | Your dedicated success manager, included |
| Make the go/no-go call | You | You | You |
The last row is the same on every route, and it should be. Our implementation guide compares the routes in full, and the timeline post shows how long each takes past the pilot.
When a pilot isn't the right next step
Skip the formal pilot if you handle a few hundred tickets a month or fewer: read every draft for two weeks and decide. Do the help desk work first if you're migrating desks this quarter. And if your desk isn't one of Zendesk, Freshdesk, Gorgias, Front, HubSpot or Intercom, Macha isn't available there; the partner ranking lists who works on other desks.
Which pilot setup fits your situation?
| You are | Pilot category | What to watch |
|---|---|---|
| A Shopify brand on Gorgias, no developer | Order status, read-only | The 7-day trial ends in build week; the 72-hour reply rule in Gorgias's billing |
| A B2B SaaS team on Intercom | "How do I" questions from the docs | Assumed resolutions; the 14-day trial ends mid-pilot |
| A small team under 1,000 tickets a month | The top category by volume | Sample size: you may need all 30 days to read 200 |
| A team that already tried native AI and switched it off | The category that failed, rescored | Write down why it failed first (rollback checks) |
Where Macha fits
Macha runs on Zendesk, Freshdesk, Gorgias, Front, HubSpot or Intercom, and its setup follows this pilot shape. With Macha's done-for-you setup, the Macha team analyzes your past tickets, recommends the first category, builds and tests the agent, and runs it in safe mode with drafts as internal notes in your help desk until your team says they're right. Its guarantee covers time, not results: live in safe mode within 7 days, or your first month is free.
The trial comes with that setup and a dedicated success manager. Its allowance covers the build and safe-mode weeks: $50 of free usage (about 125 tickets), no credit card, no time limit. The public weeks run on a paid plan. After that, the price is about $0.40 per ticket, from $299/month for 750 tickets, and it includes setup and the success manager who handles new categories, policy changes and new tools. Macha charges per ticket whether or not the agent resolves it, so it has no reason to count a resolution generously. It suits teams that want the pilot run for them and the decision left with them, and it's the wrong fit if you want to configure every rule yourself. To see the category scoring on your own queue, ask for the free support audit.
How we researched this
On October 14, 2026 we read the trial terms on Intercom's, Zendesk's and Gorgias's pricing pages, Front's Autopilot article and Fini's pricing page, plus Zendesk's resolution tiers article, Intercom's Fin outcomes article and Gorgias's pricing FAQ. Macha facts come from our done-for-you page and pricing page. The category scores and cost example are synthetic. Margins use the normal approximation at 95% confidence with p = 0.9; the zero-failure bound uses the rule of three. The plan, weights and thresholds are our recommendations, based on how our team runs safe-mode rollouts, not a published standard.
Common questions
How long should an AI customer support pilot run?
Thirty days is enough for one category: about 5 days to build and test, a week or more in safe mode, and two weeks of public replies to measure reopens and CSAT. Shorter pilots rarely read 200 tickets. Longer ones usually mean the success criteria weren't written down.
What success metrics should an AI support pilot use?
Accuracy on sampled tickets, actions outside limits, draft acceptance, escalation quality, durable resolution against your human baseline, CSAT on AI-touched tickets, and cost per resolved ticket. Use the vendor's resolution rate as a secondary number, because each vendor defines it differently.
How many tickets do you need to evaluate an AI agent?
About 200 sampled tickets for the go/no-go decision, which measures a 90% accuracy rate to within about 4 points. Read every reopen, escalation and action on top of the sample. For actions, zero failures in 300 tickets still allows a true rate of up to about 1%.
Should the AI reply to customers during the pilot?
Not at first. Start with drafts as internal notes, move part of the category to public replies once a full week of drafts meets the bar, and widen only if that slice holds. Customers shouldn't be the ones who find the agent's first misses.
What's the difference between a free trial and a pilot?
A trial is the vendor's time or usage limit; a pilot is your test plan with a baseline, guardrails and a scorecard. Trials on Gorgias (7 days), Intercom and Zendesk (14 days) end before a 30-day pilot does, so arrange the paid weeks before you start.
