Leveraging the NSSP R Studio Server to Automate QA Monitoring and Reporting

Objective: The aim of this project was to develop a nimble system to both monitor and report on the quality of Kentucky emergency department syndromic surveillance (SyS) data at system-wide and facility levels.Introduction: In 2016, the CDC funded 12 states, under the Enhanced State Opioid Overdose Surveillance (ESOOS) program, to utilize SyS to increase timeliness of state data on drug overdose events. In order to operationalize the objectives of the grant, there was a need to assess and monitor the quality of Kentucky’s SyS data, with limited resources. We leveraged the NSSP’s R Studio Server to automate quality assurance (QA) monitoring and reporting to meet these objectives.Methods: Using the R Server, we pulled data from the process messages table, aggregating messages to single patient encounters. In addition to compiling the code on a powerful remote server, the server can access the process table messages relatively quickly. We developed an R Markdown report to produce a report that includes a variety of system- and facility-level metrics that highlight key indicators of system performance and data flows. By using R, we were able to create an auto-generating QA report that runs weekly and e-mails for analyst review. Quality metrics included: % completeness of chief complaint and discharge diagnosis codes (overall and by facility)[Fig 1 & Fig 2]; visit trend by day of visit (with interactive spark lines)[Fig 2]; maximum date of message created, date message arrived at NSSP server, date of visit, and total messages[Fig 3]; message arrived trend (interactive sparklines)[Fig 3]; volume and type of error messages failing to process[Fig 4]; message volume by ADT type[Fig 5]; and volume of patient class by type by day[not shown]. Our SyS analyst reviews the report and delivers it to stakeholders with general comments about ongoing and newly emerging data quality concerns.Results: The report has proven to be beneficial in ongoing QA monitoring. The report is shared weekly with key stakeholders at the Kentucky Department for Public Health, Kentucky Health Information Exchange, NSSP, and regional ESSENCE users. Findings are reviewed at monthly SyS stakeholder meetings. The report has identified numerous errors, dead feeds, and other systems changes in near real-time; leading to corrective action and general data quality enhancement. Weekly monitoring of QA has improved data feed stability and communication of identified issue with key stakeholders.Conclusions: The R Studio Server provides a nimble platform to develop, refine, and automate a QA reporting system that can lead to improved SyS data quality. In Kentucky, in addition to improving overall data quality, these weekly reports and subsequent communication have help built relationships among key stakeholders and elevated the importance of syndromic surveillance data locally. Continual monitoring of data is critical to ensure quality and therefor the validity of the data