The rigorous coupled-wave analysis technique for describing the diffraction of multi-layer diffraction grating (MDG) is put forward. Formulations for a stable and efficient numerical implementation of diffraction efficiency are presented for MDG for TE polarization and Littrow angle. With the merit function of the -1 order diffraction efficiency higher than 96%, the parameters of MDG are optimized for HfO2 and SiO2 top layers, respectively. Numerical calculation indicates that a larger grating parameter range can be obtained with HfO2 rather than the SiO2 on the top. The manufacture tolerance and the effective incident angle for the optimized structure of MDG are discussed in the end.