Over the past half-century, according to Moore’s law, the sizes of transistors continue shrinking, and the integrated circuits have approached to their physical limits, which puts forward higher requirements for the thermal dissipation capacity of material. Revealing the physical mechanisms of heat conduction in semiconductors is important for thermal managements of devices. Experimentally, it was found that boron arsenide has a very high thermal conductivity compared with diamond, and boron arsenide has lattice constant close to silicon’s lattice constant, which can be heterogeneously integrated into silicon to solve the thermal management problem. However, group III-V boron compounds show abnormal thermal conductivities: the thermal conductivity of boron arsenide is significantly higher than that of boron phosphide and boron antimonide. Here, we use the first-principles calculation and the Boltzmann transport equation to study the thermal conductivity properties of the group III-V boron compounds. Comparison between the IV and III-V semiconductors shows that the high thermal conductivity of boron arsenide is due mainly to the existence of a large frequency gap between the acoustic and the optical branches. The energy sum of two acoustic phonons is less than energy of one optical phonon, which cannot meet the energy conservation requirements of three-phonon scattering, and then seriously restrict the probability of scattering of three phonons. The high thermal conductivity of diamond is due mainly to its great acoustic phonon group velocity. Although the boron phosphide also has a relatively large acoustic phonon group velocity, the frequency gap is relatively small, which cannot effectively suppress the three-phonon scattering, so the thermal conductivity of boron phosphide is less than that of boron arsenide. Although the frequency gap of boron antimonide is similar to that of boron arsenide, the thermal conductivity of boron antimonide is lower than that of boron arsenide due to its smaller acoustic phonon group velocity and larger coupling matrix element. The research provides a new insight into the design of semiconductor materials with high thermal conductivities.