The miscibility of quantum liquids is an interesting topic in many-body physics, which has been intensively investigated in ³He-⁴He superfluids and the mixtures of ultracold atoms. In the context of dual species Bose-Einstein condensates, the mean-field description has been well established, according to which, the miscibility condition is density independent and determined only by the ratio of inter- and intra-species interaction strength. Recently, Nadion and Petrov proposed that Phys. Rev. Lett. 126 115301, in the vicinity of the mixing-demixing threshold, quantum fluctuations play an important role to affect the equilibrium stability, and as a result, the partially miscible state emerges. This new phase of quantum matter opens up new perspectives to explore the beyond mean-field effect in ultracold atomic gases.

In this work, according to the equation of state taking the Lee-Huang-Yang correction into consideration, we investigate the ground state phase diagram of repulsive binary Bose mixtures in the interacting regime suffering a weak mean-field instability. Under the thermodynamic balance conditions, the phase boundaries between the immiscible state, partially miscible state and the homogenous state are determined. For the equal-mass case, these phase transitions only take place on condition that intra-species interactions are in an asymmetric form. In terms of interaction parameters, we explicitly derive analytical expressions of the phase boundaries, which are appropriate to describe the transitions in sufficiently dilute atomic gases. At the quantum critical point, where the partially miscible state terminates, the susceptibility tensor of the density response exhibits a divergent behavior. For the unequal-mass case, the beyond-mean-field equation of state cannot be written in a compact form, thus the determination of the phase boundaries is more involved. By expanding the Lee-Huang-Yang energy expression to the terms linear in the concentration of the minority species, we analytically obtain the threshold density for the partially miscible transition. We also propose a discriminant function, from which the configuration of the partially miscible state can be identified for the given mass ratio and interaction strength. Applications of these theoretical results to experimental systems, such as sodium, potassium, and rubidium gases, are presented.