Particle separation is important in industrial production. The granular powder exhibits the property that is like both fluid property and solid property, which makes it difficult to establish a mathematical model to reveal the particle motion mechanism. The fluid property of the granular powder can be partly explained by the classical fluid theory, but the solid property cannot be covered. Theories combining the fluid and solid properties are also used to explain the particle separation phenomenon. However, they are not in consensus about the granular theory to explain the particle separation mechanism. Friction dissipation, which represents the particle pairwise damp interactions, greatly influences the particle separation process. In order to understand the particle separation mechanism and the effect of friction coefficient on the particle motion, a three-dimensional discrete element model is used to simulate the separation of three-dimensional spherical binary particles in a cylindrical groove （shear cell）in this paper. Initially, the large particle is placed at the bottom and the other small particles pile into the groove. The shear flow of the particles is established by rotating the bottom plate of the groove. The large particles gradually jump to the top of the groove under the shearing action. The effect of particle friction coefficient on the separation is studied. Focusing on the characteristics of kinematics and dynamics in the jumping process of large particle, the influence of the friction coefficient on the trajectory, velocity and acceleration of particle are quantitatively analyzed. The conclusions are obtained as follows. 1) The process of large particle jumping can be divided into three stages: relaxation stage (the large particle stays at the bottom of the groove), the take-off stage (the large particle rises up), and the equilibrium stage(the large particle moves to the top and stays there). 2) The relaxation time decreases with friction coefficient increasing. 3) The equilibrium height of particles increases with friction coefficient increasing. 4) The amplitude of the force pulsation of the large particle increases with friction coefficient increasing. For the behavior analysis of the ascending motion of the large particle, we propose a neighborhood analysis method and define a floating factor. The ratio of the number of small particles arranged in the upper to that in the lower adjacent space of the large particle is defined as the buoyancy factor. It is found that the buoyancy factor drops sharply at the jumping point of the large particle, forming the opportunity for the large particle jumping. It is revealed that the take-off of the large particle is the result of both the high-frequency characteristics of force fluctuation and the steep drop of buoyancy factor. The rising motion of the big particle is determined by the force and the surrounding space.