MMSE-based Transceiver Design Algorithms for Interference MIMO Relay Systems

In this paper, we investigate the transceiver design for amplify-and-forward interference multiple-input multipleoutput (MIMO) relay communication systems, where multiple transmitter-receiver pairs communicate simultaneously with the aid of a relay node. The aim is to minimize the mean-squared error (MSE) of the signal waveform estimation at the receivers subjecting to transmission power constraints at the transmitters and the relay node. As the transceiver optimization problem is nonconvex with matrix variables, the globally optimal solution is intractable to obtain. To overcome the challenge, we propose an iterative transceiver design algorithm where the transmitter, relay, and receiver matrices are optimized iteratively by exploiting the optimal structure of the relay precoding matrix. To reduce the computational complexity of optimizing the relay precoding matrix, we propose a simplified relay matrix design through modifying the transmission power constraint at the relay node. The modified relay optimization problem has a closedform solution. Simulation results demonstrate that the proposed algorithms perform better than the existing techniques in terms of both MSE and bit-error-rate