The Pancharatnam phase in simple polariton model is investigated, and the influences of temperature, coupling strength, frequency difference of photon and phonon and average photon number on the evolution dynamics of Pancharatnam phase are also discussed. The results show that the Pancharatnam phase oscillates with time, the oscillating frequency and wave-form vary with time, and the variation increases with the enhancement of temperature, coupling strength and frequency difference of photon and phonon. The system Pancharatnam phase evolutes regularly when the average photon number of coherent light is small, while it becomes more chaotic as the average photon number increases.