Yb-doped fiber laser is a very appealing technology to implement space communication, laser radar and nuclear facilities due to its reduced weight, size, high efficiency, high peak power combined with good beam quality. However, Yb-doped fiber materials will suffer a harsh ionizing radiation (such as neutron, proton and electron) under the condition of irradiation. The radiation-induced darkening effect can lead the fiber loss to increase obviously, the laser slope efficiency to decrease drastically, and even no laser output to be observed in severe cases. Therefore, it is necessary to conduct in-depth research on the performance changes of Yb-doped fiber materials subjected to the irradiation. In this paper, a series of Yb-doped optical fibers and optical fibers is prepared by the improved chemical vapor deposition method combined with rare-earth chelate-doped. And the optical properties of the optical fiber before and after being irradiated and annealed are tested. We mainly investigate the absorption spectrum of Yb-doped fiber material. The results show that the concentration of Al-related defects in the Yb-doped fiber material increases after neutron irradiation, and the absorption loss in the visible region increases. And the color center defects produced by the irradiation will significantly reduce the Yb ion fluorescence lifetime. The doping of Ce ions can reduce the Al-oxygen hole center (Al-OHC) color center defects, and can suppresse the radiation-induced darkening effect of Yb-doped fiber to a certain extent. Thermal annealing can reduce the absorption of fiber material by reducing the concentration of neutron radiation-induced color center defects, and thus eliminating the darkening effect to a certain extent. Finally, with our previous research, we find that neutron irradiation and ray irradiation have similar effects on the optical properties of Yb-doped fiber materials. The main reason is that the electron ionization effects occur due to both ray irradiation and neutron irradiation to generate free electron and hole pairs, which then combine with the original defects in the material to turn into color center defects. However, the color center defects caused by neutron irradiation are more stable and require thermal annealing to be eliminated. And the results obtained in this study provide theoretical basis and solution for developing the Yb-doped fibers with high laser performance and high radiation resistance.