Since the discovery of monolayer graphene, the novel physical properties of two-dimensional (2D) materials, particularly those with fewer layers that often exhibit unique properties different from bulk materials, have received significant attention. Therefore, accurately determining the layer number or obtaining the microscopic surface morphology is crucial in the laboratory fabrication and during device manufacturing. However, traditional detection methods have numerous drawbacks. There is an urgent need for a convenient, accurate, and non-destructive scientific method to characterize the layer number and surface microstructure of 2D materials. By combining the experimental setup of laser scanning photocurrent spectroscopy, we develop a polarization-modulated scanning optical microscope based on the principle of reflectance difference spectroscopy. By monitoring the reflectivity of the samples, we can observe changes in the reflection signal strength of MoS₂ with different layer numbers. The intensity of the reflectance differential spectral signal reflects changes in the layer count within the sample. We can characterize the changes in the number of layers of 2D materials in a non-contact manner by using polarization-modulated scanning optical microscopy. Through the study of the reflectance differential spectra of two typical 2D layered materials, MoS₂ and ReSe₂, we find that our polarization-modulated scanning optical microscope system is also more sensitive to the characteristics of the stacking anisotropy of the 2D materials than the conventional reflection microscope. This indicates that our research contributes to a better understanding of the layer number characteristics and anisotropic properties of layered 2D materials. Furthermore, our research also provides a non-contact optical method to characterize the number of layers and optical anisotropy of two-dimensional layered material.