Various types of border collision bifurcations occurring in a discrete feedback transmission control protocol-random early detection （TCP-RED） system are analyzed based on normal form method. As the system parameters change, the system experiences the border collision and a sudden jump from a periodic attractor to a chaotic attractor leading to the intense vibration of the system and serious degradation of its performance. According to the linear stability condition of fixed point, a method of chaotic control is presented. With this method, the system can be stabilized to the fixed point by perturbing the state variable in the neighborhood of the fixed point. This method can significantly improve the performance and the resource utilization of TCP-RED system, and simulation results show that it is effective.