Achieving Secure Communications in Dense Multiuser Mimo Systems for 5G and Beyond

Traditional approaches for providing confidentiality of wirelessly transmitted signals are often based on cryptographic techniques. While these techniques ensure that messages are encrypted and secured against eavesdropping, they are not sufficient to protect important transmission attributes at the physical (PHY) layer. Specifically, several elds in the headers of PHY and medium access control (MAC) frames are typically sent unencrypted to maintain proper protocol functionality (e.g., sender/receiver identification). As a result, they leak side-channel information (SCI), including payload size, frequency and phase oset, modulation and coding scheme, identities of communication nodes, frame type, transmission rate, etc. These SCI can be exploited by an adversary to launch various passive (eavesdropping) and active (jamming) attacks. To complement cryptographic techniques used at upper layers and prevent the leakage of SCI at the PHY/MAC layers, PHY-layer security techniques have been introduced. These techniques exploit the characteristics of the wireless channel along with multiple-input multiple-output (MIMO) technologies, in which multiple transmitting and receiving antennas are utilized to prevent information leakage to eavesdroppers and/or avoid jamming attacks. In particular, if the signal-to-interference-plus-noise ratio (SINR) at an adversary is lower than the one at a legitimate receiver, secure communication can be achieved through the so-called wiretap coding. MIMO systems enable the transmitter and/or the receiver to generate artificial noise, called friendly jamming (FJ), so as to reduce the SINR at the adversary without impacting the SINR at the legitimate receiver. In this dissertation, we focus on exploring the security threats to next-generation wireless systems. We develop MIMO-based PHY-layer security methods to achieve reliable and secret communications in these systems. First, we consider a broadcast channel, in which a multi-antenna transmitter (Alice) sends K confidential information signals to K legitimate receivers (Bobs) in the presence of L eavesdropping devices (Eves). Alice uses MIMO precoding to generate the information signals along with her own Tx-based friendly jamming (TxFJ). Interference at each Bob is removed via a technique called MIMO zero-forcing. This TxFJ, however, leaves an eavesdropping \vulnerability region" in the proximity of each Bob, which can be exploited by a nearby Eve. Specically, because of the high correlation in the channel state information (CSI) at a Bob and nearby Eves, the TxFJ becomes less effective in preventing eavesdropping at these Eves. We address this problem by augmenting TxFJ with receiver-based friendly jamming (RxFJ), generated by each Bob. Specifically, we allow each Bob to use self-interference suppression (SIS) techniques to transmit a friendly jamming signal while simultaneously receiving an information signal over the same channel. These SIS techniques exploit circulators to attenuate the self-interference power of RxFJ at Bobs and utilize analog/digital cancellation methods to clean out any remaining residual self-interference. Considering the resulting broadcast channel, we rst study a two-user scenario and develop a secrecy encoding scheme for constructing the signals intended to two Bobs. We characterize the corresponding achievable secrecy sum-rate and formulate an optimization problem to maximize this metric. We then extend our approach to K-user scenarios and study a power minimization problem. Particularly, in our analysis we minimize the powers allocated to information and friendly jamming signals (both TxFJ and RxFJ) while guaranteeing individual secrecy rates for various Bobs. The problem is solved for the cases when the eavesdropper's CSI (ECSI) is known or unknown (but with a known distribution). Simulations show the effectiveness of the proposed solutions. Furthermore, we discuss how to schedule transmissions when the rate requirements need to be satised on average rather than instantaneously. In low and high power regime, a scheduling algorithm that serves only the strongest receivers is shown to outperform the one that schedules all receivers. Next, we consider a single-cell massive MIMO system in which a base station (BS) with a large number of antennas transmits simultaneously to several single antenna users. The BS acquires the CSI for various receivers using uplink pilot transmissions. We demonstrate the vulnerability of the CSI estimation process to a pilot-contamination (PC) attack that aims at minimizing the sum-rate of downlink transmissions by contaminating uplink pilots. We rst study PC attacks for two downlink power allocation strategies under the assumption that the attacker knows the locations of the BS and its users. Later on, we relax this assumption and consider the case when such knowledge is probabilistic. The formulated problems are solved using stochastic optimization, Lagrangian minimization, and game-theoretic methods. A closed-form solution for a special case of the problem is obtained. Furthermore, we analyze the \achievable individual secrecy rates" under PC attacks, and provide an upper-bound on these rates. We also study this scenario without a priori knowledge of user locations at the attacker by introducing chance constraints. Another attack model is considered, where the attacker generates jamming signals in both the CSI estimation and data transmission phases by exploiting inband full-duplex techniques. Our results indicate that such attacks can degrade the throughput of a massive MIMO system by more than 50%, and reduce fairness among users significantly