A self-consistent model is proposed here to study the nonlinear resonance and the hysteresis phenomena in the vertical oscillations of a charged micro-particle in a RF sheath. In this model, the charging process of the micro-particle and the sheath dynamics is considered self-consistently. And also, various forces acting on the particle are fully taken into account in the Newton's equations of the micro-particle. By solving the equation, we simulate the motions of the micro-particle in the sheath, under the excitations of the probe. Numerical results reproduce well the recent experimental observations; at the same time, we find that these nonlinearities are not only due to the structure of the sheath, but also due to the charging process of the micro-particle, ion drag force, neutral gas friction and the excitation of the probe.