The stability of nonlinear dynamical system of relative rotation is studied. Firstly, the dynamics equation of relative rotation autonomous nonlinear dynamical system with commonly damped force and forced excitation is deduced. Secondly, the stability of relative rotation nonlinear dynamical system is studied. For the nonlinear dynamical system, it is proved that the closed orbit bifurcation can occur under some conditions. Finally, The approximate solution of the equation under forced excitation is obtained by the method of multiple scales.