Fusion of Multiple Neuroimaging Modalities using Canonical Correlation Analysis

Biomedical studies frequently collect multiple measurements such as functional magnetic resonance imaging (fMRI), structural MRI (sMRI), electroencephalography (EEG), behavioral scores, and symptomatic measures from the same subject. Each modality has its own advantages and limitations, which quite often are complementary to those of other modalities. Instead of studying each dataset separately, a combined analysis promises to piece together different aspects of brain structure and function. This novel information could provide new insights to help early detection and treatment of diseases, especially diseases like schizophrenia, which impact many aspects of the brain, such as structure, function, and networks. We develop a data fusion framework to investigate the associations across complementary modalities and to find the latent sources responsible for these associations. The proposed framework is a data driven approach that identifies cross-modality associations in the form of linear correlations across modalities. We use canonical correlation analysis (CCA) to identify these correlations. This fusion framework provides a means to jointly analyze the underlying sources of correlated information from different modalities, which when analyzed separately would only provide a limited view. We utilize the proposed fusion framework to identify two types of associations: feature-level inter-subject associations and associations across simultaneously collected data. We also propose an extension of our two-modality fusion framework to fuse multiple modalities based on their associations in the form of feature-level inter-subject correlations as well as to fuse simultaneously acquired modalities across multiple subjects (at the group level). We use the multi-dataset extension of CCA, multi-set CCA (M-CCA) to identify the associations across the multiple datasets. Through the use of simulated data, we show that the proposed fusion framework provides an effective and less-constrained solution to the fusion problem. We have successfully applied the proposed methods to the fusion of a number of neuroimaging datasets: bi-modal feature-based fusion of fMRI and EEG as well as fMRI and sMRI data, multi-modal feature-based fusion of the fMRI, sMRI, and EEG, and finally fusion of simultaneously acquired fMRI and EEG data. The results show that our framework can identify cross-modality associations consistent to the task. It is also successful in identifying functional activation changes and differences in gray matter concentration due to disease. Due to their multivariate nature, the proposed methods can identify connectivity across different areas in the brain. More importantly we show that with the addition of more modalities, the specificity of these inferences increases. The findings on the simultaneous data show the usefulness of the method for identifying activation patterns in the form of amplitude modulations that are common to both fMRI and EEG data. The presented framework is flexible and can be applicable to other modalities as well as to different domains for discovering relationships among multiple types of measurements