The grain boundary-directed spinodal decomposition has a substantial effect on the microstructure and properties of polycrystalline materials. However, due to the fact that the spinodal decomposition is usually too fast to be captured in experiments, our understanding of the grain boundary-directed spinodal decomposition process is still very limited. In this work, we simulate the spinodal decomposition process of a polycrystalline system by the phase-field model, check the influences of the curvature and the atom diffusion constant inside the grain boundary (M_t) on the phase decomposition patterns, and discuss the interaction between the moving grain boundaries and spinodal decomposition. The simulation results indicate that the velocity of spinodal decomposition near the grain boundary is faster, and the spinodal morphology at the grain boundary presents the anisotropic bicontinuous microstructures different from the isotropic continuous microstructures in the bulk phase. Further, we find that the spinodal pattern is parallel to the grain boundaries with larger curvatures, and it will perpendicular to the grain boundaries with smaller curvatures. We also find that the spinodal decomposition velocity increases with the augment of M_t , while the grain boundary migration velocity will first decrease and then increase with the augment of M_t under the effect of spinodal decomposition. Finally, we simulate the spinodal decomposition process of two-grain system in three dimensions, and we obtain the results consistent with the two-dimensional simulations.