Aiming at the high requirements for illumination uniformity on the target in laser-driven inertial confinement fusion (ICF) facilities, an ultrafast smoothing method based on dynamic interference structure between beamlets of a laser quad is proposed. The basic principle of this scheme is to use a conjugate phase plate array to add the conjugate phase modulation to the multiple beamlets of a laser quad with a certain wavelength difference. Consequently, every two beamlets are coherently superposed in the far field to generate a dynamic interference pattern, resulting in the fast redistribution of the speckles introduced by continuous phase plate inside the focal spot and further improving the illumination uniformity on the target on a picosecond timescale. The coherent beamlets with a certain wavelength difference can be generated by using a broadband seed laser. Taking the laser quad of the typical ICF facilities for example, the physical model of the ultrafast smoothing method based on dynamic interference structure of beamlets is built up. The influences of the phase-plate type, the peak-to-valley value of the phase modulation and the wavelength difference between the beamlets are analyzed quantitatively, and the smoothing characteristics of the focal spot are discussed in detail and compared with those from the traditional temporal smoothing scheme such as smoothing by spectral dispersion. The results indicate that the directions of the moving speckles in the focal spot are determined by the phase-plate type. However, the required time to achieve stable illumination uniformity, i.e, the decay time, is determined by the wavelength difference between the beamlets. Moreover, the illumination uniformity on the target becomes better with the increase of peak-to-valley value of the phase modulation at first and then remains almost the same. Thus, the ultrafast smoothing method based on dynamic interference structures with well-designed phase arrays and wavelength combinations of the beamlets can realize the multi-directional and multi-dimensional speckle sweeping inside the focal spot, and further improving the irradiation uniformity on the target within several picoseconds or sub-picoseconds. Combining with the traditional beam smoothing scheme, better illumination uniformity can be achieved on an ultrashort timescale. This novel scheme can be used as an effective supplement to the existing temporal beam smoothing techniques.