A two-mode integral form projection unitary operator is introduced. Using the technique of integration within an ordered product （IWOP） of operators, the transformation property of the integral form projection operator is studied, and its normal ordered expression is derived. Then the Hamiltonian of two-mode quantum harmonic oscillator system with coordinate-momentum coupling is diagonalized by virtue of the integral form projection operator, there by the eigenenergy and eigenfunction of the system are exactly solved. Lastly, a special case is discussed.