Diamond silicon vacancy centers (SiV centers) have important application prospects in quantum information technology and biomarkers. In this work, the formation mechanism and regulation method of SiV center during the growth of polycrystalline diamond on silicon substrate are studied. By changing the ratio of nitrogen content to oxygen content in the growing atmosphere of diamond, the photoluminescence intensity of SiV center can be controlled effectively, and polycrystalline diamond samples with the ratios of SiV center photoluminescence peak to diamond intrinsic peak as high as 334.46 and as low as 1.48 are prepared. It is found that nitrogen promotes the formation of SiV center in the growth process, and the inhibition of oxygen. The surface morphology and photoluminescence spectrum for each of these samples show that the photoluminescence peak intensity of SiV center is positively correlated with the grain size of diamond, and the SiV center’s photoluminescence peak in the diamond film with obvious preferred orientation of crystal plane is higher. The distribution of Si centers and SiV centers on the surface of polycrystalline diamond are further characterized and analyzed by photoluminescence, Raman surface scanning and depth scanning spectroscopy. It is found that during the growth of polycrystalline diamond, the substrate silicon diffuses first into the diamond grain and then into the crystal structure to form the SiV center. This paper provides a theoretical basis for the development and application of SiV centers in diamond.