Optimization of Digital Coherent Transceivers for Optical Communication Systems

Coherent transceivers are the key elements of fibre-optical communication systems enabling high-speed transmission. These transceivers utilize state-of-the-art electrical and optical components, digital signal processing (DSP), and advanced coded modulation (CM) schemes. This thesis focuses on transceiver performance optimization techniques — in particular, two aspects are covered. The first aspect is related to transceiver impairments characterization and mitigation via application of advanced DSP techniques. Transceiver performance can be detrimentally affected by various imperfections in its optical and electrical components and, therefore, it is important to mitigate the impact of those imperfections. Two techniques for characterization and compensation of transceiver skews were proposed and investigated —calibration techniques based on the Gardner timing error detector and signal image spectrum measurement. Also, the impact of various transmitter in-phase/quadrature (IQ) impairments was considered—specifically, modulation impairments, skews, electrical IQ cross-talk and frequency response mismatch. Post-compensation based on advanced post-equalizer and multiple-input multiple-output (MIMO) pre-emphasis approaches were proposed and investigated. The second aspect is related to application of advanced CM techniques and optimization of the signaling scheme. Conventional uniform signaling based on quadrature amplitude modulation (QAM) is a sub-optimal solution in terms of linear and nonlinear performance, and advanced signaling schemes can be used to improve the overall system performance. Finite-length probabilistic constellation shaping, specifically, sphere shaping architecture was considered and optimized for improved linear and nonlinear performance of the system. The performance of the system employing finite-length sphere shaping architecture was extensively studied in comparison with uniform signaling and infinite-length Maxwell–Boltzmann (MB) shaping. Optimal shaping regimes were identified for long-haul multi-span and extended-reach single-span transmission links