Human vs machine: Do customer service chatbots perform better?

This study aimed to explore and compare the performance of chatbots and human agents on different tasks in online customer services. With practical Twitter data from several customer service accounts, we categorized the users’ inquiries into three categories: uncertain, complex, and ambiguous, and manually evaluated the responding posts in metrics of appropriateness, helpfulness, and empathy. Our results show that, although human agents outperformed chatbots in most tasks under different metrics, chatbots had similar performance in providing helpful responses to uncertain inquiries with a shorter response time. These results indicate that chatbots have relative advantages in responding to uncertain tasks. The findings of our study provide practical implications for chatbot design that, to minimize total transaction costs, organizations may want to consider including a module to redirect uncertain inquiries to human agents after identifying the type of the inquiry