The mode competition is the problem to be solved urgently in high-order mode gyrotron. Based on the generalized theory of transmission line, a group of first order transmission line equations that can accurately describe the coupling modes in a gradually-varying cavity is studied. Through analysing the expressions of the coupling coefficients between the various modes, a gradual-varying cavity for a gyrotron is designed. It is proved that the power in spurious modes in this novel cavity may be kept far below that in the line-joint cavity. The normalized amplitudes of non-operating modes can reach -100 dB in cavity and -88.6 dB in travelling wave guide.