In this paper, we have investigated the off-diagonal Berry phase of nonlinear systems and presented its explicit expression. The results show that, for nonlinear systems, the off-diagonal berry phase contains a new term in addition to the dynamical phase, the geometric phase and the nonlinear phase. This new term can describe a cross effect between the Bogoliubov excitation around one eigenstate and another instantaneous eigenstate, while the Bogoliubov excitations are found to be accumulated during the adiabatic evolution and contribute a finite phase of geometric nature. As an application, the off-diagonal Berry phase of a two-well trapped Bose-Einstein condensate system is calculated.