The effect of different optical magnifications on the amplitude and linearity of displacement signals measured by the photodiode quadrant detector in an optical tweezers system is studied both experimentally and theoretically. Based on the experimental results we find that the output signal with larger amplitude and more preferable linearity can be obtained when the magnified images of α particle have diameters of 50% to 60% of the detector diameter. Theoretical calculation confirms it.