By using the rigorous Fourier's modal theory and recursion algorithm of the reflection-transmission coefficient matrix(RTCM), the propagation of Hermite-Gaussian beams of different orders through gratings with sinusiodal and rectangular reliefs whose characteristic size is comparable with the wavelength, are studied and compared in detail. It is shown that the field distribution of incident beam and the structure of grating relief affect intensity distributions within the gratings. Better beam smoothing is achieved by using the rectangular-reliefed grating than using the sinusoidal-reliefed grating.