An Automated Zoom Class Session Analysis Tool to Improve Education

The recent shift towards remote education has presented new challenges for instructors with respect to teaching evaluation. Students in traditional classrooms send signals to instructors which provide feedback for the effectiveness of a given lecture. Virtual learning environments lack some of these communication channels and require new ways of collecting feedback. This work presents a suite of analysis tools for the virtual instructor. Given the transcript and video files for a Zoom meeting, this tool summarizes student sentiment and speaking characteristics. Sentiment scores are derived using state of the art Natural Language Processing (NLP) models. The video file is used to extract interesting features about the lecture content, such as the number of slides, pace of slide changes, and number of words per slide. All metrics were experimentally tested with data from four Zoom meetings, each of which included ground-truth annotations for the slide changes. Transcript-based metrics were validated by comparing to the output produced by Meeting Measures, the project this tool is based on. A time series outlier detection model was developed for the purpose of identifying slide changes during a presentation. Initially, a percentile-based model performed well on the annotated videos when the optimal threshold was known. However, the process of finding this threshold turned out to be non-trivial for videos without annotations. A Kalman filter model was then tested to alleviate the need for an optimization step. Ultimately, the percentile-based model was replaced by an HMM-based (Hidden Markov Model) model because of its ability to generalize. When tested on annotated videos, the HMM-based detector performed within a reasonable tolerance of the optimal percentile-based model. Furthermore, for each slide detected an open source Optical Character Recognition (OCR) framework was used to extract text content for computing word counts. This tool outputs a dashboard containing a set of visualizations for the instructor. These are intended to both identify pain points in the lecture as well as provide a bird\u27s-eye view of general class interaction characteristics. The broader impact goal of this work is to increase remote teaching effectiveness by helping instructors optimize education delivery throughout the academic year