Synthesis of Brain Images using Deep Learning

142 pagesDeep learning and neuroscience research have significantly progressed in the past few decades. While deep learning's conception and several key developments have been inspired by neuroscientific knowledge, the reverse osmosis of artificial neural network application in neuroscience has been limited. In this thesis, we tackle two important, but relatively under-explored tasks of brain images synthesis using deep learning. The first task bridges the gap between task-based and resting-state functional magnetic resonance imaging (fMRI), which represent the two most common experimental paradigms of functional neuroimaging. While resting-state offers a flexible and scalable approach for characterizing brain function, task-based techniques provide superior localization. We build on recent deep learning methods to create a model that predicts task-based contrast maps from resting-state fMRI scans. Specifically, we propose BrainSurfCNN, a surface-based fully-convolutional neural network model that works with a representation of the brain's cortical sheet. BrainSurfCNN achieves exceptional predictive accuracy on independent test data from the Human Connectome Project, which is on par with the repeat reliability of the measured subject-level contrast maps. Conversely, our analyses reveal that a previously published benchmark is no better than group-average contrast maps. We demonstrate that BrainSurfCNN can generalize remarkably well to novel domains with limited training data. In the second task, we apply deep learning methods in the field of neuroimaging coordinate-based meta-analysis. Neuroimaging studies are often limited by the number of subjects and cognitive processes that can be feasibly interrogated. However, a rapidly growing number of neuroscientific studies have collectively accumulated an extensive wealth of results. Digesting this growing literature and obtaining novel insights remains to be a major challenge, since existing meta-analytic tools are constrained to keyword queries. In this paper, we present \textit{Text2Brain}, an easy to use tool for synthesizing brain activation maps from open-ended text queries. Text2Brain was built on a transformer-based neural network language model and a coordinate-based meta-analysis of neuroimaging studies. Text2Brain combines a transformer-based text encoder and a 3D image generator, and was trained on variable-length text snippets and their corresponding activation maps sampled from 13,000 published studies. In our experiments, we demonstrate that Text2Brain can synthesize meaningful neural activation patterns from various free-form textual descriptions. The methodological advancements and results presented in this thesis are only the very first steps in their respective research directions but demonstrate the massive potential of deep learning in advancing neuroimaging applications and research