The d'Alembert principle and the differential equations of motion of relativistic rotational variable mass system are given. By using the invariance of the differential equations under the infinitesimal transformations, the determining equations of the Lie symmetries of relativistic rotational variable mass system are built, and the structure equation and the conserved quantities of the Lie symmetries are obtained. An example is given to illustrate the application of the result.