The modulational instability (MI) of nonlinear coherent spin waves in an atomic spin chain of spinor Bose-Einstein condensates confined in an optical lattice is studied. Analytically we obtained the criteria for the MI of nonlinear coherent spin waves and its relation with the long-range nonlinear spin coupling. We analysed the MI in the case of blue-detuned and red-detuned opctical lattice respectively. It was found that the long-range nonlinear spin coupling in the optical lattice is due to light-induced and static magnetic field-induced dipole-dipole interaction.