By studying the ultraviolet-red two-color holographic storage performances of Mg-, Zn- and In-doped lithium niobate crystals, we found that the response time of the two-color recording could be shortened by as much as 3 orders of magnitude compared to that of one-color recording, and the two-color recording sensitivity was improved significantly, which was measured to be 11 cm/J in the crystal doped with Mg of 5 mol.%. Nonvolatile holographic storage was achieved in the crystals with doping concentrations above the damage-resistant threshold value. However, gratings on the deep centers and the shallow centers were out of phase in Mg- or Zn-doped lithium niobate, while those in In-doped lithium niobate were in phase. We consider that different defects induced by different dopants are responsible for the observed results.