Generalized Lie symmetry and generalized Hojman conserved quantity of Appell equations for a variable mass holonomic system in relative motion are studied. The determining equation of generalized Lie symmetry of Appell equations for a variable mass holonomic system in relative motion under the infinitesimal transformations of groups is given. The expression of generalized Hojman conserved quantity deduced directly from generalized Lie symmetry for a variable mass holonomic system in relative motion is gained. Finally, the problem of dynamical system with three degree of freedom is studied by using the results of this paper.