The segregation of the binary granular mixture in Brazil nut problem was simulated by using discrete element model. The influences of the frequency and the amplitude of the vibration on the segregation are discussed. The simulation results show that the segregation between big and small particles can be predicted accurately. When the frequency or the amplitude of the vibration is low, the final segregation efficiency is low, but the segregation is quite stable. When the frequency or the amplitude of the vibration is high, the final segregation efficiency is still low, but the segregation is unstable. An optimum operating point with a moderate frequency and a moderate amplitude of the vibration exists in the process of segregation in Brazil nut problem.