How to measure chat support quality when you don’t have a support team
If you don’t have a support team, you don’t need a dashboard of 20 metrics. You need six numbers, checked once a week: first response time, resolution without handoff, handoff rate, customer satisfaction (CSAT), the thumbs up and down ratio on answers, and the list of questions nobody answered well. Together they tell you whether customers get answers, whether those answers are right, and what to write next.
The numbers only help if you read the conversations behind them. A 20-minute weekly routine, described below, is enough for most small businesses running an AI chatbot with human handoff. This guide explains how to calculate each metric, what can mislead you, and how to turn gaps into better answers.
Why the usual support metrics don’t fit small teams
Most support metrics were designed for contact centres: average handle time, agent occupancy, service level, tickets per agent per hour. They measure how efficiently a team of paid agents works through a queue. If your team is the owner and one colleague who answer chat between other work, those numbers describe very little.
A small business with an AI chatbot has different questions:
- Do customers get a reply quickly, including outside office hours?
- When the bot answers, is it right?
- When it can’t answer, does a person pick up in reasonable time?
- What are customers asking that we haven’t written down?
The six chat support metrics that matter
| Metric | What it tells you | How to calculate |
|---|---|---|
| First response time | Whether customers are left waiting | Median time from the customer’s first message to the first reply |
| Resolution without handoff | How much the bot handles alone | Conversations with no human involvement divided by all conversations |
| Handoff rate | How often a person is needed, and why | Handed-off conversations divided by all conversations, split by reason |
| CSAT | How customers felt about the whole chat | Ratings of 4 or 5 divided by all ratings, on a 1 to 5 scale |
| Answer thumbs ratio | Whether individual answers helped | Thumbs up divided by all thumbs (up plus down) |
| Unanswered questions | What’s missing from your content | Weekly count, plus the list of questions themselves |
1. First response time
This is the time between a customer’s first message and the first reply. Measure two versions: the bot’s first reply, which should take seconds, and the time to the first human reply after a handoff. The second one tells you about your team.
Use the median, not the average. One chat left overnight can double an average, while the median shows what a typical customer experienced.
For context, LiveChat’s customer service report gives an average first response time of 35 seconds across chats on its platform. That’s a figure for businesses with agents staffing chat, so don’t hold a two-person business to it. For human replies after handoff, a more useful target is the promise you make: if your office-hours message says “we reply within two hours”, measure the share of handoffs answered within two hours.
2. Resolution without handoff
This is the number of conversations that ended without a person getting involved, divided by all conversations. Chatbot vendors like to headline it, and it needs care. “No handoff” doesn’t mean “resolved”. A customer who got a useless answer and closed the window also counts as no handoff.
So never read it alone. Pair it with CSAT and the thumbs ratio, and read a sample of those conversations each week. A rising resolution rate with falling satisfaction means the bot is getting better at ending conversations, not at helping people.
3. Handoff rate
Handed-off conversations divided by all conversations. The total matters less than the breakdown by reason:
- The customer asked for a person. Some of this is healthy: complaints, complicated cases, people who simply prefer a human. A sudden rise may mean answers are unhelpful or customers don’t trust them.
- A keyword triggered it, such as “refund” or “complaint”. These are handoffs you chose. Check now and then that they’re still the right keywords.
- The bot wasn’t confident. It didn’t find a good answer in your content. These are mostly content gaps, and the easiest to fix.
Don’t treat a low handoff rate as the goal. In a Gartner customer survey published in 2024, the top concern about AI in customer service was that it makes reaching a person harder. A handoff that works is part of good service.
4. Customer satisfaction (CSAT)
Ask at the end of the chat how satisfied the customer was, on a 1 to 5 scale. CSAT is the share of responses that were 4 or 5, which is the formula Qualtrics describes: satisfied responses divided by all responses, multiplied by 100.
Two caveats for small businesses. First, only some customers rate a chat, and those who do tend to be the very pleased and the very annoyed. Second, samples are small. Always report the count with the percentage: “80% (12 ratings)” tells a different story from “80% (240 ratings)”. Qualtrics itself notes that benchmarking CSAT isn’t an exact science because every business is different. Your own trend is more useful than an industry figure.
5. Answer thumbs ratio
CSAT rates the whole conversation. A thumbs up or down rates a single answer, which makes it better for finding specific problems. Calculate thumbs up divided by all thumbs. Then, more importantly, read every thumbs-down answer.
Each one falls into one of four buckets: wrong content, missing content, a misunderstood question, or a correct answer the customer didn’t like (your return window is shorter than they hoped, for example). Only the first three can be fixed in the knowledge base. The fourth is a policy question, which is worth knowing too.
6. Unanswered questions
These are the questions the bot couldn’t answer from your content, or answered with low confidence. Track the weekly count and keep the list itself. The list is the most valuable output of any chatbot analytics setup, because it’s a to-do list for your knowledge base written by your customers.
Group similar questions before you act on them. “Do you deliver to Ireland?”, “shipping to Dublin?” and “EU delivery cost?” are one gap, not three.
What targets should you set?
There are few trustworthy public benchmarks for small businesses running AI chat, and vendor figures usually come from their own customers, measured with their own definitions. Rather than borrowing someone else’s number, set your own baseline: collect four weeks of data without changing anything, then choose a direction for each metric.
| Metric | Starting point | Direction |
|---|---|---|
| First response time (bot) | Seconds | Investigate anything slower |
| First human reply after handoff | The time your reply-time message promises | Keep the promise, then tighten it once you meet it consistently |
| Resolution without handoff | Your four-week baseline | Up, but only while CSAT and thumbs hold steady |
| Handoff rate | Your baseline, split by reason | Fewer low-confidence handoffs; requested handoffs roughly stable |
| CSAT | Your baseline, with the rating count | Up, or stable as volume grows |
| Thumbs ratio | Your baseline | Up, with every thumbs-down reviewed |
| Unanswered questions | Your weekly count | Fewer repeat gaps; new one-off questions are normal |
Use rolling four-week figures when volume is low. With 50 conversations a week, a single bad day can move a percentage by several points, and you’ll end up reacting to noise.
A weekly 20-minute review routine
- Minutes 0 to 3: glance at the numbers. Compare this week’s six metrics with the four-week average and note anything that moved sharply.
- Minutes 3 to 8: read every thumbs-down and every low CSAT conversation. For each, write one line: wrong content, missing content, misunderstood question, or a policy the customer didn’t like.
- Minutes 8 to 12: read five random conversations the bot handled alone. This catches confident wrong answers that nobody rated.
- Minutes 12 to 15: check handoffs. Were any left without a human reply past your promised time? Did several low-confidence handoffs share a topic?
- Minutes 15 to 20: fix the top two gaps. Write or correct the answers, then test them using the customer’s original wording.
Put it in the calendar at the same time each week. Monday morning works well, because it lets you catch up on weekend chats that arrived while nobody was watching.
How to turn gaps into knowledge
Every metric above ends in the same action: improve what the bot knows. A simple process:
- Take one question from the unanswered list or from a thumbs-down answer.
- Work out the real answer, including conditions and exceptions. If you don’t have a policy yet, this is the moment to decide one.
- Write it as a standalone answer in the customer’s words: “Do you ship to Ireland? Yes. Delivery to Ireland takes 3 to 5 working days and costs €9.95, free on orders over €80.”
- Delete or correct any older content that contradicts it.
- Test with the original question and two rephrasings.
- Check that topic again in next week’s review.
Suppose an online shop sees “can I change my delivery address?” in its unanswered list several weeks running. The owner writes one answer that covers what’s possible before dispatch, what’s possible after dispatch, and how to contact the courier. If the answer is good, that question should stop appearing among low-confidence handoffs, and the thumbs ratio on it tells you whether customers found it useful. Delivery questions like this are covered in more depth on our page for online shops.
Some tools help by listing top questions and suggesting knowledge to add based on real conversations. Treat those suggestions as drafts: the facts still have to come from you.
Getting these numbers without a spreadsheet
If your chat tool doesn’t report these metrics, a weekly export and a simple spreadsheet will do. KuraChat reports several of them directly: CSAT ratings at the end of chats, thumbs up and down on answers, analytics for conversation volume, resolution without handoff and busiest hours on the Pro plan, plus a top questions view and knowledge suggestions. See the features page for details, or compare plans on pricing.
Frequently asked questions
What is a good resolution rate for a chatbot?
There isn't a reliable universal figure, and vendors define resolution differently. Measure your own four-week baseline, then aim to raise it only while CSAT and thumbs ratings hold steady.
How many CSAT ratings do I need before the score means anything?
With only a handful of ratings, one or two responses can swing the score a lot. Report the count alongside the percentage and use rolling four-week figures when volume is low.
Should I aim for a zero handoff rate?
No. Some conversations need a person, such as complaints, refunds and exceptions. Aim to reduce handoffs caused by missing content, not the ones customers ask for.
How often should I review chatbot analytics?
Weekly is enough for most small businesses. A short weekly review catches wrong answers before they repeat and keeps the knowledge base growing from real customer questions.