According to Runge-Kutta theory, in this paper we present a new algorithm of calculating the eigenmodes of gyrotron resonators. Different from in the traditional algorithm, all properties of the eigenmodes can be determined by solving the minimum of a function with two variables, namely the resonant frequency and Q factor, in the new algorithm. Through examples of numerical calculation, it is verified that the calculation efficiency of the new algorithm is much higher than the traditional algorithm.