Bi₂Te₃-based compounds are the thermoelectric materials available only commercially, but the research on their low-temperature performances below 300 K are still insufficient. The influences of Bi/Sb ratio modulation and Se substitution on the electrical and thermal transport properties of Bi_xSb_2–xTe₃ and Bi_0.4Sb_1.6Te_3–ySe_y materials are systematically investigated in this work, aiming to optimize their thermoelectric performance in cryogenic regions through combined bandgap tuning and defect engineering. Materials are synthesized using a melt-quenching and spark plasma sintering process, and then phase analysis is conducted via X-ray diffraction and microstructural characterization by electron probe microanalysis. First-principles calculations and Hall effect measurements are used to investigate their defect formation mechanisms and carrier transport behaviors. In the Bi_xSb_2–xTe₃ system, the increase of Bi content reduces the bandgap from 0.168 eV for Bi_0.4Sb_1.6Te₃ to 0.113 eV for Bi_0.58Sb_1.42Te₃, shifting the peak ZT temperature to lower ranges. However, the enhancement of alloy scattering leads the carrier mobility to decrease from 332 to 109 cm²/(V·s) and power factor to fall from 4.58 to 1.12 mW/(m·K²). To solve this problem, Se is substituted for the Te lattice of Bi_0.4Sb_1.6Te₃. First-principles calculations reveal that the Se substitution reduces the formation energy of Se_Te + Bi_Sb complex, thus effectively suppressing Sb_Te antisite defects. This will result in the carrier concentration decreasing from 3.32×10¹⁹ to 2.64×10¹⁹ cm^–3 while maintaining high mobility at 279 cm²/(V·s). Concurrently, Se-induced point defects enhance phonon scattering, reducing lattice thermal conductivity from 0.46 to 0.38 W/(m·K), a decrease of 17%. Bi_0.4Sb_1.6Te_2.97Se_0.03 sample achieves a ZT value of 0.93 at 220 K, which is 16% higher than the pristine Bi_0.4Sb_1.6Te₃ sample with a ZT value of 0.80. The peak ZT increases from 1.17 to 1.31 at 350 K, an increase of 12%. These improvements arise from the synergistic effects of band engineering, where flattened valence band edges increase effective mass, and defect engineering, where antisite defects and strengthens phonon scattering are suppressed. This work provides a dual optimization strategy for BiSbTe-based materials, i.e. balancing bandgap reduction by controlling defects to improve cryogenic performance. The findings are particularly significant for the applications of BiSbTe-based materials in infrared detectors and multistage thermoelectric cooling systems.