This paper proposes to use minus peak time of second derivative with respect to time on logarithmic curve of temperature versus time as a characteristic time for defect depth prediction in pulsed wave thermography. First, the paper introduces the basic principle of pulsed wave thermography, and constructs the theoretical relation between logarithmic minus peak second derivative time and the square of defect depth based on the solution of semi-infinite body. Then, two specimens of steel and aluminum were manufactured with flat-bottom holes to simulate defects. Thermographic image sequences of those two specimens were obtained by using pulsed wave thermography, and then the logarithmic minus peak second derivative time were extracted. The extracted characteristic time has a very good linearity relation with the square of defect depth, and this linearity could be used for defect depth prediction in practical applications. The advantages and disadvantages of the proposed method and the widely used logarithmic peak second derivative method are discussed.