Benefitting from the high measurement efficiency, off-axis digital holography (DH) has become a most powerful DH technique for fast and high-accuracy measurement. Owing to the carrier frequency, the real image can be isolated easily in the Fourier spectrum of one off-axis hologram, so that the Fourier transform algorithm (FTA) is the most widely used algorithm for off-axis DH to realize the phase retrieval. In the FTA, one of the most important tasks is to figure out the accurate peak position of the real image and then shift the real image to the center of spectrum to remove the carrier. However, owing to the digitalization of the hologram, the peak position of the real spectrum is always not located at an integral pixel position in the practical applications, resulting in carrier residuals, thereby lowering the retrieval quality. Much work on accurately determining the peak position has been conducted to suppress the carrier residuals, such as by using the spectrum centroid method and zero padding. However, those estimation algorithms can achieve only satisfied accuracy in some situations. Then, spatial carrier phase shift (SCPS) is utilized to expand the utilization of space-bandwidth and avoid the spectrum leakage caused by band-pass filtering. The SCPS decomposes one off-axis hologram into several sub-holograms, in which the carrier induces the phase shifts between sub-holograms. Many on-axis phase retrieval algorithms are combined with SCPS to retrieve the phase from one off-axis hologram. However, the retrieved phase is usually composed of the sample phase and the carrier, so the accurate carrier information is also required to remove the carrier and obtain the correct reconstructed phase. In this paper, an accurate phase retrieval with carrier removal from single off-axis hologram by using the linear regression is proposed to achieve the simultaneous phase retrieval and carrier removal. In this method, four phase-shifted sub-holograms are extracted first from one off-axis hologram by SCPS. Since the phase shift between sub-holograms is linearly proportional to the carrier, the linear regression can be combined with least-square method to retrieve the phase and carrier simultaneously. Both the simulation and experimental results show that the proposed method can determine the carrier accurately and obtain correct phase without carrier. We believe that this proposed method can be applied to practical measurement.