A model of polarization mode dispersion (PMD) and state of polarization (SOP) variation induced coherent crosstalk is established in a polarization division multiplexing system. The properties of radio frequency power of one channel in the presence of PMD are investigated. A combined scheme of PMD compensation and polarization de-multiplexing in optical domain is proposed, which is based on monitoring of the feedback signal of RF power. A modified particle swarm optimization algorithm is also used for the adaptive polarization control. The validity of the PMD compensation and polarization de-multiplexing scheme proposed here is demonstrated in a 112Gb/s-PDM-DQPSK simulation system. Results show that the PMD tolerance of the transmission system is increased by 20 ps with 1 dB OSNR margin and the channel separation is accomplished as well.