Starting from the general transmission line equation with an electron beam, a se lf-consistent nonlinear analysis on complex cavity gyrotron with gradual transit ion is presented, in which the multiple modes interacting with an electron beam and the mode coupling are taken into account. The interaction between the electr on beam and H51-H52 RF field for third-harmonic gyrotron i s simulated numerically. The influences of multiple modes on the interaction are analyzed.