Starting from general transmission line equation with an electron beam, a self-consistent nonlinear simulation on third-harmonic complex cavity gyrotron with gradual transition is presented, in which some practical factors are taken into account. The interaction between the electron beam and- H511-H521 RF field for third-harmonic gyrotron is simulated. The influences of the thickness and the velocity spread and the velocity ratio of the electron beam and the fluctuation of the magnetic field on the multi-modes interaction with an electron beam are analyzed.