Wireless cellular networks all over the world are undergoing a profound transformation evolving from voice-oriented to data networks. Larger coverage area, better service quality, and lower energy cost are the key issues in the deployment of cellular networks. To achieve these goals, small cells, such as the femtocells and picocells, have become an important part of the current 4G and future 5G wireless cellular networks. Generally speaking, small cell networks are deployed according to the peak traffic load, which causes energy waste during low traffic periods. Against this background, energy efficiency optimization has become one of the research hotspots in wireless communications. In this paper, we focus on the energy efficiency problem in small cell networks in which a large number of small cells are spatially deployed in dense urban areas such as office buildings and shopping malls. We optimize the energy efficiency through small cell dormant mechanism under the constraints of average connection ratio (ACR) and average downlink channel capacity. First, we derive the mathematical expressions for average downlink channel capacity and ACR in three-dimensional (3D) small cell networks by Poisson point process (PPP) theory. Second, the monotonicities of channel capacity and ACR are analyzed in detail. Then, based on the results of monotonicity analysis, the optimal small cell dormant probability is calculated to satisfy the constraints of ACR and average downlink channel capacity respectively. Finally, we formulate a network energy consumption minimization problem subject to the constraints of ACR and channel capacity to determine the dormant probability. In addition, we also formulate an optimal maximum connection number of small cells, which minimizes the energy consumption subject to the joint constraints of ACR and channel capacity. Numerical results show that our 3D PPP model is more accurate than the traditional two-dimensional (2D) one in both channel capacity and ACR performance, and that the energy consumption of small cell networks can be reduced by about 21% of the total energy consumption with the dormant strategy in this paper. More importantly, the optimal dormant probability and appropriate configuration of the maximal number of connection can be effectively used to design small cell dormant strategy for 3D small cell networks.