Phase drift caused by environment is the main source of quantum bit error rate and instability in the double M-Z interferometer quantum key distribution(QKD) system. The phase drift consists of both high-frequency and low-frequency vibrations, the former, which is due to environment noise, is usually conducted via the atmosphere and can be isolated by technological measures (e.g. using foamed-plastics wrapping or rubber pillow ), the latter comes from building-vibration, which vibrates slowly with a serious impact on M-Z interferometer, with a period of 3±0.32 min (min level). A novel real-time tracking and compensation method for phase drift caused by low-frequency vibration is proposed. With the improved scheme, the QKD system based on double M-Z interferometers can operate stably for 24 hours, and key exchange with an error below 6% over 75 kilometers has been achieved in the lab.