In this paper, we systematically investigate the dynamics of non-magnetic spheres (polystyrene spheres) and magnetic nanoparticles dispersed in Fe3O4 magnetic colloid under an externally applied magnetic field. It is found that the polystyrene spheres form chain-like structures when the direction of magnetic field is parallel to the sample cell. The whole dynamic process of polystyrene spheres in the magnetic field can be characterized by a fast interaction between polystyrene spheres and magnetic nanoparticles and a slow interaction among polystyrene chain-like structures respectively. When a magnetic field is applied in the direction perpendicular to the sample cell, polystyrene spheres can be assembled into a short-range ordered two-dimensional structure due to the repulsive interaction among polystyrene spheres. Once the applied magnetic field excesses a critical level, a flower-shaped complex structure can be formed due to the attractive interaction between the polystyrene sphere and the magnetic cluster.