We have studied various complex motions of the irregular dust grains immersed in non-uniformly magnetized plasma. The cylindrical magnet that we used for experiments significantly alters the radial distribution of the sheath potential which confines the negatively charged grains. Grains are horizontally illuminated by a 50 mW, 532 nm laser sheet and imaged by a CCD camera from the upper transparent electrode. Hypocycloid and epicycloid motions of grains are observed for the first time as far as we know. Cuspate cycloid motions, circle motion, wave motion, and stationary grains are also observed. Their trajectories can be obtained by using long-time exposure, and the characteristic parameters of the grain movement are measured by using the image processing with MATLAB. Though the dust grains can move around the magnet steadily in various trajectories, the induced magnetic field is too weak to give rise to cycloid motions of grains. Then we propose a new mechanism that an inverse Magnus force induced by the spin of the irregular grains plays an important role in their cycloid motions. The pollen pini we used for experiment is not a regular microsphere, there is a symmetry in the shape. On the basis of Bernoulli principle, the pressure difference between the left and right side of the forward moving grains produces the inverse Magnus effect. Additional comparison experiments with regular microspheres are also performed to confirm that the cycloid motions are distinctive features of an irregular dust grain immersed in the plasma. The periodical change of the cyclotron radius as the grain travels would result in the (cuspate) cycloid motions, and the maximal value of angular velocity of spin is about 105 rad/s. Our experimental observations can be well explained based on the force analysis in 2D horizontal plane.