Abstract: The influence of dispersive magnetic permeability on propagation of untrashort pulses in metamaterials is mainly in that it leads to the appearance of the pseudo-χ(5), self-steepening (SS) and second-order nonlinear dispersion terms in the propagation equations. In this paper, the role of dispersive magnetic permeability in modulation instability (MI) in metamaterials is identified based on the Drude model. It is found that in the anomalous dispersion regime, the pseudo-χ(5) nonlinear parameter, which is always negative, increases the MI frequency and growth rate, which is opposite to that in ordinary positive-index materials; the SS tends to suppress MI regardless of its sign, while the second-order nonlinear dispersion effect tends to stimulate MI in the positive-index region and suppress MI in the negative-index region. In the normal dispersion regime, in which MI cannot occur in the ordinary materials, MI can occur due to the role of the second-order nonlinear dispersion, suggesting a new way of generating solitons or ultrashort pulse trains in the normal dispersion regime.

Keywords: 
• metamaterials /
• modulation instability /
• dispersive magnetic permeability /
• second-order nonlinear dispersion