The low-energy structure (LES) of above-threshold ionization (ATI) of atoms subjected to an intense laser field is a hot topic in the strong-field atomic physics. The rich physical insights behind LES attract a lot of attention. Based on a semi-classical model, a semi-classical two-step (SCTS) quantum trajectory model and numerical solution of the time-dependent Schrödinger equation (TDSE), we study the pulse-duration dependence of LES for Xe atom subjected to a mid-infrared laser field. It is found that the energy of LES becomes lower for shorter pulse duration. Further analysis shows that in the case of multi-cycle laser field, the LESn structure is closely related to the number of times of forward scattering and the initial transverse momentum. In the case of few-cycle laser pulse, the carrier-envelope phase (CEP) dependence of the peak position of LES is mainly due to the CEP dependence of the influence of both vector-potential of the laser field and the Coulomb potential. In addition, the bunching effect of electrons, caused by Coulomb potential, is the main reason for the formation of LES.