In order to meet the low insertion loss of transmission band and the low reflection loss of reflection band in detection technology of millimeter and sub-millimeter wave, in this paper, we design a frequency selective surface (FSS) filter based on waveguide array structure in the sub-millimeter wave band. Through optimizing the structural parameters, the FSS filter overcomes the drawback that the insertion loss increases at large angle of incidence. Meanwhile the FSS filter is not sensitive to the change of angle of incidence, these make it satisfy the requirements of particular engineering. In addition, the frequency responses of the FSS filter to TE and TM are the same, so it can detect the dual-polarized radiation signals simultaneously. Firstly, we give initial values of structural parameter according to the indicator. Secondly, we simulate and optimized the structure by simulation software. Finally, a new FSS filter with a low insertion loss is given. Simulation results show that the FSS filter meets the system requirements and overcomes the shortcoming of large insertion loss at a large incident angle. Then, we analyze the main source of the insertion loss in transmission band. An error sensitivity analysis of structural parameters of FSS filter is also given, which provides a reference for fabrication.