A technique for direct measurement of group delay of optical elements is introduced. With the joint time-frequency analysis of white-light spectral interferogram, group delay can be directly extracted from the ridge of wavelet-transform. The technique is accurate and simple. The measurement results of group delay and group delay dispersion of a piece of fused silica was demonstrated. The results agree well with those from theoretical calculation, and the noise is greatly reduced. This technique is suitable for various application of white-light interferometer.