Detection of a signal in colored noise: A random matrix theory based analysis

This paper investigates the classical statistical signal processing problem of detecting a signal in the presence of colored noise with an unknown covariance matrix. In particular, we consider a scenario where m-dimensional p possible signal-plus-noise samples and m-dimensional n noise-only samples are available at the detector. Then the presence of a signal can be detected using the largest generalized eigenvalue (l.g.e.) of the so called whitened sample covariance matrix. This amounts to statistically characterizing the maximum eigenvalue of the deformed Jacobi unitary ensemble (JUE). To do this, we employ the powerful orthogonal polynomial approach to determine a new finite dimensional expression for the cumulative distribution function (c.d.f.) of the l.g.e. of the deformed JUE. This new c.d.f. expression facilitates the further analysis of the receiver operating characteristics (ROC) of the detector. It turns out that, for m=n, when m and p increase such that m/p is fixed, there exists an optimal ROC profile for each fixed signal-to-noise ratio (SNR). In this respect, we have established a tight approximation for the corresponding optimal ROC profile