Because of high surface-to-volume ratio (SVR), the most prominent size effect limiting thermal transport originates from the phonon-surface scattering in nanostructures. Here in this work, we propose the mechanism of phonon scattering by the under-coordinated atoms on surface, and derive the phonon scattering rate of this mechanism by quantum perturbation theory combined with bond order theory. The scattering rate of this mechanism is proportional to SVR, therefore the effect of this mechanism on phonon transport increases with the feature-size of nanostructures decreasing. Due to the ω⁴ dependence of scattering rate for this mechanism, the high-frequency phonons suffer a much stronger scattering than the low-frequency phonons from the under-coordinated atoms on surface. By incorporating this phonon-surface scattering mechanism into the phonon Boltzmann transport equation, we calculate the thermal conductivity of silicon thin films and silicon nanowires. It is found that the calculated results obtained with our model are closer to the experimental data than those with the classical phonon-boundary scattering model. Furthermore, we demonstrate that the influence of this phonon-surface scattering mechanism on thermal transport is not important at a very low temperature due to the Bose-Einstein distribution of phonons. However, with the increase of the temperature, more and more phonons occupy the high-frequency states, and the influence of this scattering mechanism on phonon transport increases. It is astonished that the phonon scattering induced by the under-coordinated atoms on surface is the dominant mechanism in governing phonon heat transport in silicon nanostructures at room temperature. Our findings are helpful not only in understanding the mechanism of phonon-surface scattering, but also in manipulating thermal transport in nanostructures for surface engineering.