On the outlying eigenvalues of a polynomial in large independent random matrices

Second version. Comments are welcome.Given a selfadjoint polynomial $P(X,Y)$ in two noncommuting selfadjoint indeterminates, we investigate the asymptotic eigenvalue behavior of the random matrix $P(A_N,B_N)$, where $A_N$ and $B_N$ are independent Hermitian random matrices and the distribution of $B_N$ is invariant under conjugation by unitary operators. We assume that the empirical eigenvalue distributions of $A_N$ and $B_N$ converge almost surely to deterministic probability measures $\mu $ and $\nu$, respectively. In addition, the eigenvalues of $A_N$ and $B_N$ are assumed to converge uniformly almost surely to the support of $\mu$ and $\nu,$ respectively, except for a fixed finite number of fixed eigenvalues (spikes) of $A_N$. It is known that almost surely the empirical distribution of the eigenvalues of $P(A_N,B_N)$ converges to a certain deterministic probability measure $\eta$ (sometimes denoted $\eta=P^\square(\mu,\nu)$) and, when there are no spikes, the eigenvalues of $P(A_N,B_N)$ converge uniformly almost surely to the support of $\eta$. When spikes are present, we show that the eigenvalues of $P(A_N,B_N)$ still converge uniformly to the support of $\eta$, with the possible exception of certain isolated outliers whose location can be determined in terms of $\mu,\nu,P$, and the spikes of $A_N$. We establish a similar result when $B_N$ is replaced by a Wigner matrix. The relation between outliers and spikes is described using the operator-valued subordination functions of free probability theory. These results extend known facts from the special case in which $P(X,Y)=X+Y$