A series of Al-Ni-RE (RE=La, Ce, Y) amorphous ribbons were prepared. The crystallization behavior and primary phases were investigated using X-ray diffraction and differential scanning calorimeter, and their relationships with the alloy composition and atomic features were analyzed. Results show that if the topological instability parameter λ was modified using effective atomic radii, the metallic glasses in an Al-Ni-RE system can be classified into nanocrystalline, nanoglassy and glassy types according to their crystallization behaviors by two critical values of the modified topological instability parameter λ'. Linear relationships of onset temperature of crystallization and mixing enthalpy with λ' are also clearly revealed, indicating that λ' is closely related to the thermal stability of Al-Ni-RE metallic glasses.