Inferential Statistics and Information Theoretical Measures: An Approach to Interference Detection in Radio Astronomy

In a time when technology is rapidly growing, radio observatories are now able to expand their computational power to achieve higher receiver sensitivity power and a more flexible realtime computing approach to probe the universe for its composition and study new astronomical phenomena. This allows searches to go deeper into the universe, and results in the recording of massive quantities of observed data. At the same time, this increases the amount of radio frequency interference (RFI) found in the obtained observatory data. The high power of RFI easily masks the low power of extraterrestrial signals, making them hard to detect and potentially blinds radio telescopes from parts of the universe. Between the influx of RFI and the massive amounts of data recorded per second, a need for robust, efficient, high-performance, real-time RFI characterization and flagging algorithms has become abundantly clear. While there are current RFI characterization and flagging algorithms already implemented specifically for Radio Astronomy, many are not openly published and are fine tuned to the specific application a radio telescope is being used for, i.e hydrogen mapping, “fast radio burst” (FRB) detection, or transient and pulsar searches. Often the current RFI detection algorithms are implemented after post-processing has been completed. This means that data can be lost which could increase RFI signal strength while further decreasing the signal strength of celestial signals, making them harder to detect. We work with the “rawest” form of observatory data engineers have access to, complex-valued channelized voltages and their respective high resolution power spectral densities. As a baseline or “ground truth” we use Median Absolute Deviation (MAD) to detect RFI in complex-valued channelized voltages and Spectral Kurtosis (SK) to detect RFI in power spectral density data. In a new perspective, we use inferential statistics and information theoretical measures to formulate new real-time RFI detection algorithms for both multi-domain (time-frequency) data formats. Using inferential statistics, we apply the Shapiro-Wilk Test for Normality on complex-valued channelized voltages. We then apply information theoretical measures by extracting the Differential Spectral Entropy and Spectral Relative Entropy of both complex-channelized voltages and power spectral density data. The baseline RFI excision algorithms are compared against our novel algorithms to determine how effective and robust the new interference detection algorithms are