Recent studies showed that the nominal AlN interlayers in InAlN/AlN/GaN heterostructures had high GaN mole fractions, especially those grown by metalorganic chemical vapor deposition. The Al and Ga mole fraction in the AlGaN interlayer determine the electron wave function and penetration probability, and thus affecting the scattering mechanism related to the InAlN/AlGaN potential layers. In this paper we study the effects of Al mole fraction of the AlGaN interlayer on three scattering mechanisms related to the potential layer, i.e. alloy disorder scattering, subband energy fluctuation scattering and conduction band fluctuation scattering induced by In compositionally inhomogeneous InAlN layer. The wave function and penetration probability in the InAlN/AlGaN/GaN heterostructure are determined by self-consistently calculating the Schrödinger-Poisson equations and then used to calculate the scattering mechanisms. The results show that penetration probabilities in the InAlN and AlGaN both decrease with increasing Al mole fraction. The combination of the contribution of the screening effect and the two-dimensional electron gas (2DEG) density inhomogeneity results in an initial decrease and subsequent increase in the subband energy fluctuation scattering-limited mobility with increasing Al mole fraction, and the heterostructure with a smaller InAlN thickness has a larger mobility increase. The penetration probability and non-periodic arrangement of Al and Ga in the AlGaN predict an Al mole fraction dependence of the alloy disorder scattering-limited mobility similar to the subband energy fluctuation scattering-limited mobility, and the alloy disorder scattering occurs mainly in the AlGaN because the penetration probability in the AlGaN is much higher than in the InAlN. The conduction band fluctuation scattering-limited mobility monotonically increases with increasing Al mole fraction due to the decrease of the penetration probability. The subband energy fluctuation scattering-limited mobility is less sensitive to variation in the Al mole fraction than the other two scattering mechanisms-limited mobilities. In a small Al mole fraction range around 0.1, the alloy disorder scattering is a dominant scattering mechanism, while the subband energy fluctuation scattering dominates the mobility beyond this compositional range. When Al mole fraction is above 0.52, the three scattering mechanisms-limited mobility exceeds that in the InAlN/GaN heterostructure without the AlGaN interlayer, indicating the promotion of the mobility by the AlGaN interlayer. The mobility is raised by more than 50 percent in the InAlN/AlN/GaN heterostructure with an AlN interlayer compared with that in the InAlN/GaN heterostructure without the interlayer.