Data challenges in pulsar searches

Technological advances coupled with a decline in digital storage costs have resulted in a profusion of data being created, collected and consumed. These data give rise to new challenges and opportunities in many disciplines ranging from science and engineering to biology and finance. An example of a future project in radio astronomy that promises both Big Data and Big Discoveries is the Square Kilometre Array (SKA) radio telescope project. Astrophysicists are confident that the Big Data amassed by the SKA will not only answer fundamental questions regarding the Universe but also contain big discoveries not yet postulated. The transformational potential of the SKA and its ensuing data and algorithmic challenges, in particular for the discovery and study of pulsars, drive the research of this thesis. Discovering all pulsars beaming towards Earth is one of the key science goals of the SKA. However, in addition to low signal strengths, searching for pulsars is extremely difficult due to the intrinsic weakness of their signals, propagation effects and the presence of anthropogenic interferences. Numerous techniques have been developed to overcome some of these difficulties and to assist in the quest to find more pulsars. However, despite the success of these techniques, the number of pulsars discovered in recent surveys (Swiggum et al. 2014, Lazarus et al. 2015) has fallen well short of the number predicted by pulsar population synthesis models (Lorimer 2011). This shortfall in pulsar detections can be attributed to radio frequency interference (RFI), red noise and scintillation (Lazarus et al. 2015). For this thesis, and in order to investigate and quantify these claims, I first developed a new technique to simulate pulsar search data that contain different types of RFI and varying noise baselines (i.e. red noise). This surrogate modelling technique was then used in a framework that I developed to inexpensively explore the sensitivity of pulsar search pipelines for different noise and RFI settings. The results from this framework highlight the necessity to develop algorithms that are able to identify and remove non-stationary variations from the data before RFI excision and searching is performed in order to limit false positive detections. To address the shortcomings identified with the framework which assessed the performance of existing pulsar search pipelines, I developed a new real-time algorithm for excising RFI while simultaneously normalising the variability in time and frequency inherent to pulsar observations. Processing synthetic data with the algorithm resulted in an expansion of the noise/pulsar spin period parameter space for which we are able to successfully detect pulsars. Furthermore, the algorithm is shown to reduce the number of false positive detections. In conclusion, the insights gained from the work presented in this thesis and the improvements achieved will contribute to the development of a new realtime pulsar search pipeline adept at dealing with the challenges posed by the SKA.