Optical diffractive neural network (ODNN) uses light wave as a computing medium to perform the inference and prediction function of neural network, which has the advantages of high speed, low power consumption, and parallel processing. In this work, an ODNN with only three layers of phase modulation is designed, and a method to improve the recognition performance of ODNN based on the first-order spectral distribution of targets is proposed. Using this method, the parameters of a three-layer ODNN are effectively optimized and the optimal pixel size, diffraction distance, and method for image preprocessing are obtained. The three-layer ODNN designed in this work has a recognition accuracy rate of 95.3% for MNIST (handwritten digit set), compared with the 91.75% accuracy achieved by the five-layer ODNN in the reference (Lin X, Rivenson Y, Yardimci N T, Veli M, Luo Y, Jarrahi M, Ozcan A 2018 Science 361 1004). In addition, the system volume is greatly reduced and the system structure is simplified. Combined with the advantages of high speed and low power consumption, it has huge potential applications in the fields such as edge computing in the future.