Dicke model, as an important many-body model in quantum optics, describes the interaction between multiple identical two-level atoms and a quantized electromagnetic field. This spin-boson model shows collective phenomena in light-matter interaction systems and can undergo a second-order quantum phase transition from a normal phase to a superradiant phase when the coupling strength between the two-level atoms and the optical field exceeds a critical value. Dicke model embodies unique many-body quantum theories. And it has been widely studied and obtained many significant research results in quantum information, quantum process, and other quantum systems. Meanwhile, Dicke model also has wide applications in quantum optics and condensed matter physics.

The extended Dicke model, describing the interaction of a Bose-Einstein condensate in an optical cavity, provides a remarkable platform for studying extraordinary quantum phase transitions in theory and experiment. Based on the recent experiment on non-Hermitian coupling between two long-lived atomic spin waves in an optical cavity, in this work we use spin-coherent-state variational method and present the macroscopic quantum-state energy of the non-Hermitian Dicke model.

The spin coherent state variational method has an advantage in the theoretical research of macroscopic quantum states, especially in the normal and the inverted pseudospin states. In the variational method, optical coherent states and atomic extremum spin coherent states are used as the trial wave functions. A Hermitian transformation operator is proposed to diagonalize the non-Hermitian Hamiltonian, which is different from the ordinary quantum mechanics where the transformation operator must be unitary. Herein, the energy function is not necessarily real in the entire coupling region. Beyond an exceptional point, the spectrum becomes complex and introducing biorthogonal sets of atomic extremum states is necessary to evaluate the average quantities.

The normal phase (for the zero average photon number) possesses real energy and atomic population. The non-Hermitian interaction destroys the superradiant phase (for the stable nonzero average photon number) and leads to the absence of quantum phase transition. However, the introduced atom-photon interaction, which is induced by the pump field experimentally, can change the situation, dramatically. The pump field can balance the loss by the non-Hermitian atom-photon interaction to achieve the superradiant phase.

An interesting double exceptional point are observed in the energy functional. There is the real spectrum below the first exceptional point and beyond the second exceptional point, while there is a complex spectrum between these two exceptional points. The superradiant phase appears only beyond a critical value, which is related to the nonlinear interaction and the pump laser. A new and inverted quantum phase transition from the superradiant phase to the normal phase, is observed by modulating the atom-field coupling strength. The superradiant phase of the population inversion state appears for a negative effective frequency and a large atom-photon interaction. The influence of the dissipative coupling may be observed in cold atom experiment in an optical cavity. All the parameters adopted in this work are the actual experimental parameters.