The geometric phase of two-level atom, under the non-Markovian effect of environment, is investigated from the microscopic Hamiltonian. The results show that the geometric phase of atom strongly coupled with cavity field is larger than that of atom weakly coupled with cavity field, and the difference between them becomes larger as the loss of environment increases. In the case of relatively small environment loss, the geometric phase of atom shows continuous or discontinuous evolution with time depending on different initial atom states, and the discontinuity range increases when the environment loss increases. In a word, the geometric phase exhibits complicated characteristics under non-Markovian environment.