A deep learning approach to wireless system design for channel sensing, contention & estimation

Deep Learning techniques are expected to play a key role in the development of wireless systems at the Physical (PHY) and Medium Access Control (MAC) layer for sixth generation (6G) communication networks. In particular, learning-based advancements would be needed to provide for (a) more efficient utilization of shared spectrum to accommodate an ever-increasing number of wireless devices and (b) improved scalability of existing signal processing techniques as the spatial and frequency dimensions of wireless architectures rapidly expand. In the first part of this dissertation, we propose a multi-agent deep reinforcement learning (RL) framework to perform contention-based medium access in shared spectrum. Centralized approaches to spectrum sharing require excessive real-time overhead messaging and obtaining the solution is known to be NP-hard. Instead, we assume that base stations operate in a fully decentralized fashion, and model shared spectrum access of base stations performing spectrum sensing as a decentralized partially observable Markov decision process (MDP). We introduce a two-stage MDP in each time slot that uses information from spectrum sensing and reception quality to make a medium access decision. Our distributed reinforcement learning framework achieves performance competitive with a genie-aided adaptive energy detection threshold. We then extend this framework to a larger action space of medium access and transmit modulation scheme. We modify the reward function to provide for this extension and utilize a stabilizing reinforcement learning technique to provide for scalability, hence achieving an improved cumulative reward on both indoor and outdoor layouts with a large number of BSs. In the second part of this dissertation, we develop a deep generative learning framework to perform channel estimation from an insufficient number of pilots in high dimensional wireless systems. Channel estimation using generative priors assumes that the reconstructed channel lies in the range of the generative model, and optimizes the input vector to estimate the channel matrix. We show that our approach outperforms state-of-the-art compressed sensing (CS) baselines. Subsequently, we develop a novel over-the-air design for training the aforementioned deep generative models using Generative Adversarial Networks from pilot measurements instead of clean channel realizations, and still achieve performance competitive with the CS baselines.Electrical and Computer Engineerin