Physical layer security in emerging wireless communication systems

Security has become an increasingly significant and urgent issue in wireless networks. The growing computational capability of the eavesdropper (Eve) and more and more complicated secure key management in large scale and heterogeneous networks, pose more and more stringent requirement to the cryptographic protocols. As a new design paradigm, physical layer security has been recognised as a promising solution to enhance the security of wireless links relying on the channel properties and advanced signal processing, and can act as either an alternative or a complementary solution to the conventional cryptographic methods. Without complicated secure key generation and management, physical layer security is quite suitable for the large scale distributed networks. In this thesis, we aim to improve the secrecy performance in emerging wireless communication systems relying on different physical layer security techniques. Firstly, the security of an amplify-and-forward successive relaying network with multiple untrusted relay nodes is investigated, where the conventional detrimental inter-relay interference is exploited to jam the untrusted nodes without assistance of external helpers. Considering different complexity requirements, several relay selection schemes are proposed, and the closed-form expressions of the lower bound of the secrecy outage probability and the maximum secrecy diversity order are derived accordingly. It is shown that the maximum secrecy diversity order of N-1 can be achieved for an N relay nodes network. Secondly, the artificial noise (AN) aided secure transmission strategy for a multiple-input single-output multiple-Eve (MISOME) system with a secure user (Bob) and a normal user (NU) is investigated. The power allocations among Bob, NU and AN, as well as the wiretap code rates, are jointly optimized to maximize the effective secrecy throughput (EST), under the average throughput constraint of NU. An alternative optimization algorithm with guaranteed convergence is proposed to obtain the optimal parameters. It is shown that the EST increases with the transmitting power and the number of transmit antennas, and decreases with the throughput constraint of NU, and the EST can be improved through injecting AN and concurrent transmission of Bob and NU. Thirdly, physical layer security in a multi-antenna small-cell network is investigated, where the multi-antenna base stations (BSs), cellular users, and Eves are all randomly distributed according to independent Poisson point processes. Stochastic geometry is applied to analyze the connection and secrecy outage probabilities and the average achievable secrecy rate. The impact of different parameters, including power allocation, BS and Eve density, and the adaptive eavesdropping or jamming on the secrecy performance is analyzed. It is shown that the average secrecy rate is a quasi-concave function of the power allocation factor and monotonically decreases with the ratio of the Eve-BS density. Finally, we study the physical layer security in a large-scale heterogeneous network consisting of both sub-6 GHz massive multi-input multi-output (MIMO) macro cells and millimeter wave (mmWave) small cells. By considering pilot spoofing attacks from the Eves, the coverage and secrecy probabilities are derived using stochastic geometry and the conditions under which the millimeter wave tier outperforms the sub-6 GHz counterpart are discussed in terms of both coverage and secrecy.It is shown that the mmWave small cell can provide better coverage performance in the high transmission code-rate region, and the secrecy performance of the mmWave system outperforms the sub-6 GHz counterpart in the low redundant rate region, which reveals the advantage of using mmWave for secure communication.Doctor of Philosophy (EEE