The characteristic basis function method is known as an effective method to solve the electromagnetic scattering problems, but the convergence of the iterative solution of the reduced matrix equation is slow when the characteristic basis function method is used to analyze the electromagnetic scattering characteristics of the electrically large target. In order to mitigate this problem, a new reduced matrix construction method is proposed to improve the iterative solution efficiency of characteristic basis function method in this paper. Firstly, the singular value decomposition technique is used to compress the incident excitations, and the characteristic basis functions of each sub-domain under the new excitations are solved. Then, the new excitations and the characteristic basis functions are defied as the testing and basis functions to construct the reduced matrix. The diagonal sub-matrices of the reduced matrix constructed by the new testing and basis functions are all identity matrices, thereby improving the condition of reduced matrix. Thus, the total number of iterations to achieve reasonable results is significantly reduced. Numerical simulations are conducted to validate the performance of the proposed method. The results demonstrate that the efficiency of the iterative solution of the reduced matrix equation constructed by the new method is significantly improved. Furthermore, the characteristic basis functions’ generation time required by the proposed method is noticeably less than that by the traditional characteristic basis function method due to the reduced number of matrix equation solutions.