In a radio frequency identification system, designing a tag anti-collision algorithm with good performance is the key to improving the system throughput. In this paper, we propose a principle of dynamic frame size regulation and a method of tag grouping, and design a novel algorithm called grouped dynamic framed slotted ALOHA (GDFSA) through combining the above, which first estimates the number of unread tags and divides them into groups, then uses dynamic frame slotted ALOHA in each group. The simulation results show that the proposed algorithm can stabilize the throughput rate between 34.6%-36.8% and the time slots increase linearly in the identification procedure. When the tag number is beyond 2000 compared with conventional algorithms, the proposed algorithm can enhance the slot efficiency by 30%, and optimize the throughput rate obviously and save the system resource.