A novel composite-channel AlxGa1-xN/AlyGa1-yN/GaN high electron mobility transistor has designed and optimized. The influence of the two-dimensional electron gas and electric field on device structure parameter is obtained from the self-consistent solution based on the theory of semiconductor energy band and quantum well. The influence of the layer structure of the device on its performance is obtained from simulation using TCAD software. Combining the results of theoretical analysis and simulation, the optimized structure Al0.31Ga0.69N/Al0.04Ga0.96N/GaN HEMT is proposed. The simulation results show that the device with gate length of 1 μm and gate width of 100 μm has the maximum transconductance of 300 mS/mm and small fluctuation in the gate voltage of from -2 V to 1 V, which shows the excellent linearity of the device. The maximum current density of 1300 mA/mm, the cut-off frequency of 11.5 GHz and a maximum oscillation frequency of 32.5 GHz are obtained.