The development and application of random matrix theory in adaptive signal processing in the sample deficient regime

Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy at the Massachusetts Institute of Technology and the Woods Hole Oceanographic Institution September 2014This thesis studies the problems associated with adaptive signal processing in the sample deficient regime using random matrix theory. The scenarios in which the sample deficient regime arises include, among others, the cases where the number of observations available in a period over which the channel can be approximated as timeinvariant is limited (wireless communications), the number of available observations is limited by the measurement process (medical applications), or the number of unknown coefficients is large compared to the number of observations (modern sonar and radar systems). Random matrix theory, which studies how different encodings of eigenvalues and eigenvectors of a random matrix behave, provides suitable tools for analyzing how the statistics estimated from a limited data set behave with respect to their ensemble counterparts. The applications of adaptive signal processing considered in the thesis are (1) adaptive beamforming for spatial spectrum estimation, (2) tracking of time-varying channels and (3) equalization of time-varying communication channels. The thesis analyzes the performance of the considered adaptive processors when operating in the deficient sample support regime. In addition, it gains insights into behavior of different estimators based on the estimated second order statistics of the data originating from time-varying environment. Finally, it studies how to optimize the adaptive processors and algorithms so as to account for deficient sample support and improve the performance. In particular, random matrix quantities needed for the analysis are characterized in the first part. In the second part, the thesis studies the problem of regularization in the form of diagonal loading for two conventionally used spatial power spectrum estimators based on adaptive beamforming, and shows the asymptotic properties of the estimators, studies how the optimal diagonal loading behaves and compares the estimators on the grounds of performance and sensitivity to optimal diagonal loading. In the third part, the performance of the least squares based channel tracking algorithm is analyzed, and several practical insights are obtained. Finally, the performance of multi-channel decision feedback equalizers in time-varying channels is characterized, and insights concerning the optimal selection of the number of sensors, their separation and constituent filter lengths are presented.Academic Programs Office at WHOI, the Office of Naval Research (ONR Grants N00014-05-10085, N00014-07- 10738, N00014-09-10540, N00014-10-10259 and N00014-11-10426), and the National Science Foundation (NSF Grant ECCS-1102156