A car must be repaired and inspected. Furthermore, a car has to pass the TÜV in order to be able to continue driving. What chatbot testing has to do with cars and the TÜV, you will learn in this article!
Most probably share the experience. You go to a website, want to ask something, and a chatbot pops up. Typically, there are two reactions a chatbot can trigger in you. First, a chatbot can lead to great frustration if it doesn't understand the request or understands it incorrectly. One is justifiably frustrated because the request was not answered and one has to describe it again via email or phone. On the other hand, the chatbot can answer a request correctly. Here, it takes the work out of searching through the website to find the right information.
😡 "The chatbot doesn't understand my request."
😊 "I got the right answer within a few seconds without searching for a long time.
A bad chatbot - three characteristics by which you can immediately recognize it
What is a bad chatbot? For this, we first need to understand the basic concept of how chatbot technology works. AI chatbots work with so-called confidence scores. Confidence scores range from 0 to 1 and indicate the percentage of the chatbot that has recognized an intention. For each question asked by the user, the chatbot calculates a confidence score and attempts to give the correct answer to the user on this basis, using probabilities. The confidence scores are usually preset to between 0.7, which means that the chatbot only outputs answers if the probability of recognition is at least 70%. Let's assume, for example, that the user asks a question that the chatbot recognizes at 50%. In this case, the chatbot would respond with its standard answer that it did not understand the statement. If the recognition rate of a chatbot is set too low, e.g. to 50%, the chatbot will understand the question but give an incorrect answer.
(1) The chatbot does not understand the user's request. Even after repeated rephrasing and different sentence structures, the chatbot cannot answer the query. This is often because the bot has too little training data.
(2) The chatbot gives the wrong answer. Another common phenomenon is that the chatbot outputs the wrong answer to a question. The result is clear, the user:s feel misunderstood. The basic building block of communication is to feel understood. Accordingly, chatbots must be able to conduct a genuine dialog and fully understand the question.
(3) The chatbot has too little content. A well-known problem of non-quality chatbots is that they have too little content. What is meant here is that the most important customer queries are not mapped in the chatbot. This occurs when the bot is not aligned with the company's experience or support requests.
Chatbot Testing - for qualitative bots
The team behind melibo has been creating AI chatbots for 3 years now. Through the experience we have found some learnings that will immediately improve your chatbot testing and lead to significantly better results.
Define use case correctly
What do you want your chatbot to stand for? What are your most important customer queries? Good chatbot testing starts with the chatbot content and the right focus.
70/30 approach: you cover 70% of the queries with the chatbot and you do not answer exotic queries. This framework ensures that your focus in chatbot testing is not on the complicated queries, but on the recurring standard queries. Typically, 90% of the users query just 10% of the topics of your chatbot. Accordingly, concentrate on the really important topics and train the most frequent chats.
Understanding the Customer Journey
What are the most important points of your product/service that need to be explained? At what point do your users need help?
In the customer journey, you need to understand which cycle your users are in and what questions they have. Typically, your customers arrive at your website via marketing, recommendations or by chance. As a new customer, they naturally have questions about the product/service and want advice. With the chat flows from melibo you can navigate your users safely to their destination and increase your conversions. During chatbot testing, it is especially important that you check your flows for plausibility and completeness. Do your users understand your product/service better through the chatbot? As a tip: try to build up the customer journey retrospectively. What leads to a successful lead and a purchase? Check your chats for the most important points and provide quality customer advice.
Chatbot testing by the target group
You know your users best. After you have identified the target group that will use the chatbot and its characteristics, you can start building the chatbot. Based on the target group, think about the questions your users could ask the bot. With this in mind, you create the chatbot content that will ultimately determine its quality.
Who asks better questions than your users? No one. As simple as it sounds, the best training data and content comes from your target audience. Accordingly, you should try to get this data through test phases. Create the chatbot according to your ideas and validate the result against your target audience. In this way, you test whether the requests deviate from your ideas and content. In the end, you ensure that the bot is really trained for the requests of your users and enhances the user experience.
Surveys for targeted chatbot testing
Building on point 3.), you can also launch surveys on social media or via e-mail. Use surveys on LinkedIn to find out which content and topics are important to your customers. A relevant point in chatbot testing is completeness. Do you capture the most important requests of your users? You can use a survey to ask specific questions and check how your users would use your chatbot. Do they need help with a product purchase, discount promotions or would you like to process returns faster? The survey helps you understand if your project is on the right track.
Chatbot testing through internal test phases
There are different ways to ask for a concern. Accordingly, it helps when testing the chatbot if your chatbot goes through an internal test phase. Ask employees or other contacts from your environment if they can ask your chatbot questions. Set a framework for this and indicate which topics the chatbot can answer. In this way, you define the areas in which questions can be asked. You will quickly notice that different questions/statements are made on one topic. Your chatbot testing will benefit from the fact that you collect data and can cover different question variations in the bot.
Insights and Bot Gym
One of the key points in chatbot testing is the phase after you put your chatbot online. Regardless of the test phases and the setup, you will only find out in practice whether and how your chatbot is received. We recommend that you don't wait too long with the go-live, but improve your AI chatbot based on practical training data. Especially in the first weeks after the go-live, you should regularly check your insights and see which questions your chatbot can't answer. Everything you need to know about chatbot training here.
Summary - Chatbot Testing
In this article, we gave you six tips to improve your chatbot testing and showed you how to build your bot qualitatively. It's important to note that you don't have to follow all the tips for your chatbot to be operational. In fact, chatbots learn the fastest via practice data. So don't hesitate too long to go live and benefit from hands-on testing.
We hope you enjoy the ride!
