Based on the dynamic equations of particles system in non-inertial coordinate system, the higher order differential variational principle of relative motion for mechanical system is obtained, and the energy of higher order relative velocity of the mechanical system is introduced, the different kinds of higher order dynamic equations of holonomic mechanical system in relative motion are derived, and an example is given to illustrate the application of the results.