A thermodynamic model that describes the thermal properties of the compound phases in the nanocrystalline （NC） alloy systems has been developed，which provides quantitative predictions on the stabilities and transition features of different phases. By using the NC Sm-Co alloy system as an example，the temperature dependence of the mole Gibbs free energy of the NC alloy phases with different excess volumes in the nano-grain boundary regions were provided. Based on the model calculations，the relative stability of different phases and the phase transition rules in the NC Sm-Co system were analyzed. It was shown by the calculation results that near the room temperature，some of the NC alloy phases，whose mole Gibbs free energy values change from negative to positive due to the decrease of the nano-grain size，will transform into stable NC alloy phases. This is distinctly different from the phase transformation features in the coarse-grained alloy systems，whose thermodynamic properties are only dependent on the temperature. The experimental results on the phase configuration and the phase stability obtained from the prepared NC Sm-Co alloy confirmed the theoretical predictions of the present thermodynamic model.