Six months in: what to read in your AI agent's monthly report

In the first few months with an AI agent, you read the reports with a magnifying glass. Every escalated conversation gets reviewed; every odd reply gets discussed. By month six, the opposite happens: the report arrives, the numbers are green, you open it for thirty seconds and file it. Right when the agent has become part of the business is when it gets looked at least.

This piece is for you if you've had an agent handling customers for six months or more and the monthly report has turned into a formality. By the end you'll know which five things to look at, which signs look good but aren't, and what the report will never show you however carefully you read it. If you're after the quick Monday check-in, that's a different thing and it's here; this is the long view, once a month.

What changes after month six

Early on, the risk was that the agent would get things wrong. At six months the risk is different: the business has changed and the agent hasn't.

A lot happens in half a year. You've raised a price, dropped a service, taken on someone new, switched to summer opening hours. Each of those decisions was made in a meeting or a corridor chat, and not all of them made it into the agent's playbook. The monthly report won't flag that directly; you need to know where to look.

1. Escalation rate: be wary when it drops too far

The escalation rate is the share of conversations the agent hands to a person. It falls in the first few months, and that's a good sign: the agent picks up your answers and you fill in its playbook.

But if it's still falling at month six, ask why. It might genuinely be getting better. Or it might be answering things it used to escalate and shouldn't: a complaint it dismisses with a generic apology, a pricing question answered with last year's rates.

The check is simple: take five conversations from the month that the agent resolved on its own, on topics it used to escalate. Read them in full. If all five are fine, great. If one isn't, you know what to adjust.

2. The "couldn't answer" list

Most agents log the questions they couldn't answer. At the start that list is long and varied. By month six it should be short — and what matters is whether anything new appears.

Compare it with last month's. A question that wasn't there before and now shows up four times nearly always points to a change in your business the agent doesn't know about: a new product, a promotion someone announced on Instagram, roadworks that change how people get to you. It's the cheapest sign that the playbook has gone stale.

3. Compare blocks, not single months

A month with 20% more conversations doesn't say much on its own. It could be August, a campaign, or one big customer writing thirty times.

With six months of data you can compare quarters: the last three months against the first three. Has volume grown? Is response time holding? Are the topics the same, or has the mix shifted? Three-month trends are far more reliable than month-to-month jumps, and they stop you worrying about noise.

4. When the playbook was last updated

This one usually isn't in the report, and it's among the most useful. When were the agent's playbook, the price list it reads from, or the FAQ answers last touched?

If the answer is "month two", you have a problem even if every number is green. Put a date on it: a playbook review every quarter, with one named owner and a list of what's changed in the business since last time.

5. Hours and cost, with real numbers

By month six you can answer the question that justified the project: how many hours is it saving, and what does it cost? If you haven't worked it out, here's a simple method.

Focus on cost per resolved conversation, not the total bill. If volume has grown, the bill goes up, and that's fine. What shouldn't go up is what each well-handled conversation costs you.

What the report won't show you

Three things no monthly report captures well, and it's worth knowing so you don't get complacent:

• Customers who left without complaining. A conversation abandoned halfway shows up as "unresolved", if at all. You don't know whether they got bored, phoned instead or went to a competitor.
• The people who never wrote in. If someone prefers to ring because they don't trust the chat, the agent never knows.
• Tone. A report counts replies, not whether they sounded like your business. You only see that by reading conversations.

So reading the report needs to go hand in hand, at least once a month, with reading five or six random conversations. Ten minutes. No metric replaces it.

When you don't need all this

If your agent handles a few dozen conversations a month, a monthly report is mostly noise: one customer writing three times shifts the percentages. In that case, do this review quarterly and spend the time reading conversations rather than charts.

And if the report you get includes neither the escalation rate nor the unanswered questions, that's not on you: ask your provider for them. The system already has that data.

The five questions, to keep handy

Is the escalation rate falling for good reasons? Are there new unanswered questions? What do the last three months say against the first three? When was the playbook last updated? What does each resolved conversation cost?

Answer all five each month and the agent stays yours, rather than something running in the background that nobody quite understands. And if, after six months, you can see another task that could be handed off, the agent catalogue shows what else you can take off your plate on the same terms: you approve, they execute.