Nd2Fe14B rare earth and Sm2Co17 type permanent magnets have been widely used in the third generation of synchronous radiation light source and free electron laser facility in undulators and other components of particle accelerators. In addition, the permanent magnets are used in the radiation treatment system for cancer as a beam line component. Compared with Sm2Co17 type permanent magnet, Nd2Fe14B rare earth permanent magnet has the characteristics of large magnet energy product, rich starting materials and low price. Although its Curie point and coercive force are lower than those of Sm2Co17 type of permanent magnet, Nd2Fe14B rare earth permanent magnet is still widely used. As an important part of the accelerator, the magnetic loss phenomenon appears when permanent magnet is used in long-term irradiation environments, which affects the stability and quality of the beam. Therefore, it is important to investigate the magnet demagnetization induced by photon irradiation. Recently, there have appeared many researches of the phenomena of demagnetization for the permanent magnets under the irradiation of various kinds of particles. By using different research methods and experimental conditions, single particle irradiation is performed and then the effect of irradiation on magnetic loss is investigated by comparing the macro magnetic properties (such as magnetic flux loss rate, saturation magnetization, etc.). However, there are not any available reports on the microstructure investigations of permanent magnets after irradiation. Microstructure affects macroscopic magnetic properties. In order to discuss the microscopic demagnetization mechanism, the transmission electron microscope is used to characterize and analyze the microstructure evolutions of Sm2Co17 type permanent magnet and Nd2Fe14B rare earth permanent magnet before and after proton irradiation. The evolution of the number density of nanocrystal and its size distribution induced by proton irradiation are calculated. Moreover, the effect of microstructure evolution on macroscopic magnetic loss is discussed. The results indicate that the microstructure of permanent magnet transforms from single crystal structure to polycrystalline structure with the increase of the proton irradiation damage level. Nanocrystal and the matrix of permanent magnet have the same crystal structure. With the irradiation damage level increasing, the nanocrystal density of Nd2Fe14B first increases and then decreases, while the particle size distribution first increases and then keeps constant; the number density of nanocrystal of Sm2Co17 type permanent magnet gradually decreases, while particle size gradually increases, and comparing with Sm2Co17 type permanent magnet, the crystal structure of Nd2Fe14B permanent magnet shows an obvious tendency to be amorphous in 2 dpa irradiation damage level.