This paper described a high-order mode oscillation phenomenon, in the simulation of four-cavity intense relativistic klystron amplifier(IRKA). And we have analyzed the cause of this kind of oscillation, and explored the suppression methods for the whole tube. Two-dimensional and three-dimensional electromagnetism simulation software are used to study the influence of parameter’s changes (such as the oscillation frequency of cavities, the cavity Q value, the length of drift tube, and the characteristic impedance) on the high-order mode oscillation. The effectiveness of the high-order mode oscillation suppression methods is verified by simulation. Then, by optimizing the structure of IRKA, we have achieved an output microwave with a power of 2.13 GW, a gain of 60 dB and an efficiency of 26% in simulation, and the high-mode oscillation is avoided effectively. Finally, an output microwave with a power of 1.9 GW, a gain of 61 dB, and an efficiency of 24% is gained in the experiment.