Improving Channel Equalization with Neural Network and Reinforcement Learning

In wireless communications, transmitted signals suffer from distortion caused by the channel. Equalization is used to mitigate such effects. However, a receiver needs to be adaptable to account for many types of channel effects, some of which may be nonlinear. Neural networks have been proven to be an effective method for wireless channel equalization due to their ability to learn and solve complex problems. In this thesis, two different techniques are presented to improve the use of Artificial Neural Networks (ANN) such that they can perform channel equalization more effectively. For the first technique, we present a system consist of multiple neural network (NN) equalizers, each trained on a specific channel such that a signal can still be equalized regardless of changes in channel conditions. In the first part of this experiment, we test the performance of this system of neural network equalizers on arbitrary channel models (i.e. squared, cubic, etc.). In the second part, we investigate the effectiveness of the system of neural network equalizers for different High Frequency (HF) channel conditions. The output of each NN equalizer in our proposed system is combined and optimized to select the best-equalized signal. Past research in the literature indicates that NN equalizers are vulnerable when channel conditions change because NN equalizers are trained on specific channel conditions. Based on simulation results, we show that our system can learn which NN equalizer is best suited for a particular channel as the channel varies over time. For the second technique to improve the channel equalization performance, we proposed using reinforcement learning to tune the hyperparameters of a neural network equalizer. When a neural network is created, before it can be deployed, a process called hyperparameter tuning is required for a neural network to perform at its best at a given application. For this work, we used an annealing epsilon greedy algorithm, which is a reinforcement learning technique to tune different attributes of a neural network equalizer. Reinforcement learning has been used to tune neural networks for other applications, but to the best of our knowledge, it has not been done for neural network equalizers. HF is also the assumed channel for this part of our work, and we investigate the effectiveness of using the annealing epsilon greedy algorithm to tune a neural network equalizer by comparing its equalization performance with a fixed neural network equalizer (i.e. by fixed we mean that the neural network can learn the weights but cannot change the value of its hyper-parameters). From the results obtained, at three distinct HF channel conditions used for this work, the neural network equalizer tuned with the epsilon greedy algorithm can outperform a fixed neural network equalizer