GaInAsSb quaternary alloys have attracted much interest in infrared optoelectronic applications due to their versatility in a large range of energy gaps from 0.296 eV to 0.726 eV when lattice matches to GaSb wafer. However, due to the high intrinsic carrier concentration and Auger recombination, GaInAsSb p-n junctions typically are characterized by high dark current density at room temperature and need to be operated at low temperature to obtain high optoelectronic performance. In this work, a front surface wide-bandgap semiconductor nano pillar array (NPA) and a high reflective metal back surface reflector (BSR) are designed to modulate optoelectronic performances of GaInAsSb p-n junction. The optical and optoelectronic characteristics are analyzed by the finite difference time domain simulation and the numerical solution of carrier transport equations, respectively. It shows that the NPA-BSR structure can trigger Mie-type resonance, Wood-Rayleigh anomaly effect and Fabry-Perot resonance, which can be used to trap the light efficiently in an ultrathin GaInAsSb film. Owing to these nanophotonic effects, the average light absorption of ~90% can be obtained in 1.0–2.3 μm infrared waveband for 1μm Ga_0.84In_0.16As_0.14Sb_0.86. It also shows that the Auger recombination can be suppressed with thickness decreasing which leads the carrier collection efficiency to increase and the dark current density to decrease. Theoretical results show that the carrier collection efficiency of ~99% and dark current density of ~5×10^–6 A/cm² can be obtained for the 1 μm Ga_0.84In_0.16As_0.14Sb_0.86 p-n junction. With these unique optoelectronic properties, the NPA-BSR nanophotonic structure can become a very promising method to realize the high performance ultrathin GaInAsSb infrared optoelectronic devices.