Turning Data into Insight: Analyzing Chat Transcripts to Revitalize Virtual Reference Services

As academic libraries transition to tiered reference services, public service points are relying more heavily on the work of student assistants. In a library where chat services are monitored by a team of graduate student assistants, there is a concern about the quality of service being provided. Chat transcript data offer a valuable lens through which to access the content and assess the quality of online reference transactions. This presentation reports a study of analyzing reference chat transcripts data to evaluate chat reference service quality and performances of graduate student assistants in an academic library. Anonymized chat transcripts were analyzed because they provided an objective and non-intrusive source of data to observe and evaluate students\u27 performances. A rubric was designed to help us break down chat performance into measurable skill sets such as assessing patron information need and quality of information resources offered. It also provided a framework to rate these skills on a scale from poor to excellent. Over the course of the project, we analyzed over 750 chat transcripts from the work of eight graduate assistants from two semesters. We classified transcripts by patron question type and analyzed both overall and individual chat performances. Ultimately, we used the findings of our data analysis as insight to revise chat reference training, implement new procedures, and retool our supervision strategies for online reference services