Maximum Likelihood Sequence Estimation in the Presence of Timing Misalignment and Polarization Mode Dispersion in Optically Amplified Return-to- Zero Systems

Abstract—We investigate the performance of maximum likelihood sequence estimation (MLSE) receiver in the presence of the impairments from both the pulse carver-data modulator timing misalignment (TM) and polarization mode dispersion (PMD) in optically amplified return-to-zero (RZ) systems. RZ modulation format is commonly used in long-haul wavelength-division multiplexing transmission systems and the dominating noise source in those systems is amplified spontaneous emission (ASE) noise, which is signal dependent and requires special study when direct detection is employed. In this paper, based on the bit-to-bit error probability estimation using Karhunen-Loeve (KL) expansion and decorrelation of noise components, we use the steepest descent method to obtain sequence-to-sequence error probability and achieve the performance evaluation of MLSE receiver with arbitrary input signal pulse shape, optical filtering and electrical filtering taken into consideration. Monte Carlo simulations of a 10-Gb/s RZ system are demonstrated and agree with the theory well. The results show that the power penalty induced by TM and PMD can be effectively reduced by MLSE receiver, which thus validates its capability to enhance tolerance to both TM and PMD with shared electrical devices. Keywords- Equalizers, optical fiber communication, modulation I