Benefitting from the high measurement efficiency, off-axis digital holography (DH) has been becoming the most powerful DH technique for fast and high-accuracy measurement. Due to the carrier frequency, the real image can be isolated easily in the Fourier spectrum of one off-axis hologram, so that the Fourier transform algorithm (FTA) is the most widely used algorithm for off-axis DH to realize the phase retrieval. In FTA, one of the most important tasks is to figure out the accurate peak position of the real image and then shift the real image to the center of spectrum to remove the carrier. However, due to the digitalization of the hologram, the peak position of the real spectrum is always not located at an integral pixel position in the practical applications, resulting in carrier residuals to lower the retrieval quality. Much work has been conducted to estimate the accurate peak position to suppress the carrier residuals, such as spectrum centroid method and zero padding. However, those estimation algorithms can only achieve satisfied accuracy in some situations. Then, spatial carrier phase shift (SCPS) is utilized to increase the utilization of space-bandwidth and avoid the spectrum leakage caused by band-pass filtering. SCPS decomposes one off-axis hologram into several sub-holograms, in which the carrier induces the phase shifts between sub-holograms. Many on-axis phase retrieval algorithms have been combined with SCPS to retrieve the phase from one off-axis hologram. However, the retrieved phase is usually consisted of the sample phase and the carrier, so the accurate carrier information is also required to remove the carrier and get the correct reconstructed phase. In this paper, an accurate phase retrieval with carrier removal from single off-axis hologram by using linear regression is proposed to achieve simultaneous phase retrieval and carrier removal. In this method, four phase-shifted sub-holograms are firstly extracted from one off-axis hologram by SCPS. Since the phase shift between sub-holograms is linear proportional to the carrier, linear regression can be combined with least-square to retrieve phase and carrier simultaneously. Both the simulation and experimental results show that the proposed method can determine the carrier accurately and obtain correct phase without carrier. We believe that this proposed method can be applied in practical measurement applications.