There are many ways to measure the effectiveness of customer service automation. Some metrics are “Automation Rate” and “Ticket Deflection”. But what do those terms really mean, and are they the most effective way to measure things?
Automation rate
A tech provider claims they can automate 60% of tickets - sounds pretty good! But another claims just 30%. Is that only half as good?
The problem is that there is no industry standard definition of automation rate. Let’s take a step back and say, what do we even mean by automation?
The chances are, most tickets have a degree of automation to them. Sending an auto-reply to inform the customer that you have got their request and will deal with it within, say, 24 hours – that’s a form of automation! You could easily get to 100% automation that way.
How about if your customer service bot determines what your customer is asking and sends them to an FAQ page? That’s a higher level of sophistication, but is it a good experience for the customer? It depends if, when you ask, the customer says that you’ve solved their problem.
You can cherry-pick tickets with some level of automation, but it doesn’t speak to whether the automation actually accomplished anything.
Ticket Deflection
When considering ticket automation, we must also address deflection. Deflection means reducing the number of customer service tickets that hit your team. This is typically achieved through having a self-service tool or portal where customers can find answers. It could also mean a knowledge base, FAQ, and more recently, chatbots.
It sounds great in theory - fewer tickets that your team have to deal with is surely a good thing. However, it can be hard to measure how much deflection you get. Just by having a delivery policy on site, you are deflecting tickets because people who find it won’t ask as many questions!
But once again, the question remains - has the customer’s question been resolved? Do you ask every customer visiting your FAQ pages whether they found what they needed? Do you know for sure that a customer who interacted with your chatbot didn’t then speak to an agent?
The golden metric - Ticket resolution rate
Ultimately, what do we care about? Solving customers' problems, not just deflecting them. The quicker, cheaper, and more thoroughly we can do that, the better.
That’s why we advocate ticket resolution rates when it comes to measuring customer service automation. How many tickets did the chatbot or automation tool actually solve, without any human intervention, vs. how many did it attempt to solve?
How do we measure that? If the chatbot has to hand a ticket over to a human agent, it has yet to solve the problem. If the chatbot gets to the end of a conversation and asks the customer, “Did I solve your problem?” and the customer says, “Yes, " it’s a success. Or if a particular CSAT score has been met at the end.
(We sometimes use automation rate as an alternative because that’s what we’ve always called it internally, but it is misleading.)
Think of it this way:
Automation rate measures how busy your automation tool is, but not how good the work is
Deflection measures how much work it claims to have done, regardless of whether it actually did anything.
Ticket resolution measures the quality of the work that is being done.
Conclusion
If you are speaking to someone who uses Automation Rate or Ticket Deflection as a success metric, ask them these questions:
How do you define a ticket that has been automated, vs. one that has not?
Does automation rate include any tickets that have not been fully resolved?
How do you define ticket deflection? Or, how do you measure how many tickets are deflected?
Can you be sure that a ticket which has been “deflected” has found the answer they were looking for?
To be clear - these metrics are perfectly valid, but they are prone to misuse and cherry-picking. If the business has a clear methodology for measuring these metrics, then it’s a legitimate measurement. But be aware that there isn’t a standardised metric, and the numbers may not reflect the true value that is being provided.
Talk to our team to learn more about how you can use customer service automation to fully resolve tickets.
