How to read an AI agent demo without buying the hype

Four minutes into the screen share, an entire customer journey has happened. A message comes in, the agent replies, checks the calendar, offers two slots, books one and sends the reminder. No errors, no awkward pauses, not a single "sorry, I didn't catch that". You come off the call with the uneasy feeling that everyone except you already has this running. That feeling is the output of the demo, not evidence about the software. So let's look at the same screen differently: what the thing you just watched actually proves, which words need translating, and which questions turn a showroom into something you can decide on.

A demo proves that the demo works

Start from the assumption that nobody is lying to you. They're showing you the happy path — the only route that fits in fifteen minutes and the only one you can rehearse.

On the happy path, the customer writes in tidy, complete sentences. They say "I'd like an appointment on Thursday afternoon." They don't type "hi do u have anything free" at 11:40pm and then vanish for two days. They don't send a fifty-second voice note with the telly on in the background. They don't ask about a service you stopped offering in March. They don't say "yes, Thursday" and then, two messages later, "actually Friday's better, and can you do my sister too".

Your real customers do all of that constantly. That's precisely where a tool either holds up or falls over gracefully.

Which makes the single most useful request in any demo a short one: "can I type?" Take the keyboard and write what one of your customers would write at eight on a Friday evening. A vendor who trusts their product hands it over without blinking. If you're told that particular bit is "still in the test environment", you've learned something worth knowing.

Three words worth translating

"Agent." Right now almost everything gets called an agent, so the label carries no information. The question that does: does it act, or does it only talk? A real agent takes actions in your tools — writes to the calendar, creates the customer record, sends the email, updates an order status. If all it does is draft text that somebody on your team then copies into your actual system, it isn't an agent, it's a very fast copywriter. That may still be worth paying for, but the saving is the writing, not the admin, and the price should reflect that.

"Integrates with your calendar / CRM / shop." "Integrates" covers everything from a genuine two-way, real-time connection to "we export a CSV overnight" or "there's an automation you could wire up yourself". Three questions settle it in about a minute: does it read and write, or only read? How often does it sync? And if someone moves an appointment by hand on their phone, does it notice?

"It learns on its own." Be careful with this one. What you want isn't a system that changes by itself — it's a system that changes when you change the rules. So ask it concretely: "if it told a customer the wrong thing this morning, what exactly do I do so it doesn't repeat that tomorrow, and how long before the change takes effect?" If the answer is "it learns as it goes", read that as "you're not the one in control".

Four awkward questions (and why they work)

1. "What happens when it doesn't know?" The most important of the four. A system that always produces an answer is worse than one that says "let me confirm that and come back to you" and flags it to you. Ask to see it live with an edge case from your business — the kind that comes up once a month.
2. "Show me something that went wrong." Ask for a real client where it misfired and what they did afterwards. If nothing has ever gone wrong across their entire book of customers, either there's no book or they aren't telling you.
3. "Who approves, and where do I see it?" Anything that leaves your business as a commitment — a price, a lead time, a quote — should pass your desk before it goes out, at least for the first few weeks. Then the other half of the question: if I'm away for three days, does everything stop, or are there rules that let it carry on without me?
4. "What do I take with me if I leave?" Conversations, customer records, the rules you've written. If walking away means losing the history of your own customers, that isn't a supplier — it's a door that only opens inwards.

None of these is hostile. They're exactly the questions a good vendor welcomes, because they head off a customer who'd have been furious two months in.

The demo isn't what decides the purchase

A demo is for ruling things out, not for choosing. Choosing needs a small pilot, and "small" means four things written down before you start:

• One channel. WhatsApp, or the website form, or email. Not all three at once.
• One kind of task. FAQs, or bookings, or status updates. Whichever eats the most of your week.
• A short window. Two or three weeks is enough to see the pattern.
• A measurable definition of "this is working". What share of messages get resolved without you, how fast it replies out of hours, how many replies you had to correct.

That last one is what almost nobody writes down in advance, which is why arguments about whether it's working end up being an exchange of impressions. Write it on day one, even on the back of an envelope.

And set a review date. A pilot without an end date isn't a pilot — it's a subscription nobody feels brave enough to cancel.

When this much scrutiny is overkill

Now the other side. Not everything deserves this level of suspicion. If all you need is something to answer opening hours, the address and whether there's parking, most decent tools on the market will do, and interrogating one like it's an ERP purchase wastes an afternoon you don't have. Scrutiny should scale with what the thing can break: the closer it gets to prices, deadlines, customer data or money, the more awkward questions it earns.

The reverse trap is worth naming too. A dull demo isn't automatically an honest one, and plenty of good products come with enthusiastic salespeople. What separates the two isn't tone — it's whether they let you touch it.

Before your next call

Save three real customer messages — the difficult ones, with typos and a change of mind halfway through — and bring them to the demo. It's the best evaluation tool you'll ever have and it took five minutes to assemble.

This applies to us as well: our demos show the happy path, like everyone's. If you're weighing up something like a receptionist agent to run WhatsApp, your website and email from one place, ask us for the keyboard and put the four questions to us. Especially the second one.