Modulation instability (MI) is a fundamental nonlinear phenomenon. Considering that parity-time symmetric non-Hermitian periodic potentials and higher-order nonlinearities are currently active research directions in optical systems, in this study we investigate the nonlinearity-induced modulation instability of optical beams propagating in parity-time symmetric non-Hermitian periodic potentials. Eigenstates of this system are spatially extended Bloch waves. Through linear stability analysis combined with the other method of nonlinear propagation evolution, we systematically investigate the modulation instability of Bloch waves in the lowest band of a parity-time (PT) symmetric non-Hermitian periodic potential in the presence of cubic or quintic nonlinearities, respectively. We demonstrate the distinct characteristics of modulation instability in self-focusing and self-defocusing nonlinear regimes.
First, regardless of cubic or quintic nonlinearity, an important feature arising from the PT symmetric non-Hermitian periodic potential is that the nonlinear Bloch band becomes asymmetric with respect to the center of the Brillouin zone, which in turn leads to an asymmetry of the Bloch wave modulation instability about the Brillouin zone center.
Second, for self-defocusing and self-focusing cubic nonlinearities, the regions in the Brillouin zone where modulation stability and instability occur for Bloch waves on the energy band are complementary to each other. The mechanism underlying the instability is the collision of two bands among the collective excitation spectrum in the eigenvalue equation of the linear stability analysis.
Third, regardless of whether the quintic nonlinearity is self-defocusing or self-focusing, it renders the Bloch waves across the entire Bloch band modulationally unstable. In the self-defocusing quintic nonlinear case, a new instability mechanism emerges: the presence of discrete energy level crossings in the collective excitation spectrum. This mechanism introduces new features to the instability, including a small instability amplitude and its existence only at discrete excitation quasimomenta. The presence of the level-crossing instability mechanism in the self-defocusing quintic nonlinear case is another important feature of PT symmetric non-Hermitian periodic potentials.
All results from the linear stability analysis can be verified through propagation evolution simulations of the Bloch waves. The research findings enhance the understanding of the influence of higher-order nonlinearities on modulation instability in PT symmetric non-Hermitian periodic potentials, and also provide a theoretical foundation for potential applications of nonlinear control in non-Hermitian optical propagation.