The total population of atoms (ρ_<em>a</em> = 1 − 2ρ_<em>b</em>) in the ground state as a function of the transition strength <em>J</em> for different particle numbers

<p><strong>Figure 3.</strong> The total population of atoms (ρ_<em>a</em> = 1 − 2ρ_<em>b</em>) in the ground state as a function of the transition strength <em>J</em> for different particle numbers. The red solid line is the MFA result (i.e. <em>N</em> → <b>∞</b>).</p> <p><strong>Abstract</strong></p> <p>The quantum phase transition in an atom–molecule conversion system with atomic transition between two hyperfine states is studied. In the mean-field approximation, we give the phase diagram in which the phase boundary only depends on the atomic transition strength and the atom–molecule energy detuning, but not on the atomic interactions. Such a phase boundary is further confirmed by calculating the fidelity of the ground state and the energy gap between the first excited state and the ground state. As a comparison to the mean-field results, we also study the quantum phase transition with the full quantum method where the phase boundary depends on the particle number. Analysing the finite-size scaling behaviours of the energy gap, the fidelity susceptibility and the first-order derivative of entanglement entropy with respect to the atomic transition strength, we obtain their critical exponents by numerical calculation and show that in the thermodynamic limit, one can obtain the same phase boundary as in the mean-field approximation. Our results show a new way to manipulate the quantum phase transition by regulating the atomic transition strength with the intensity of the laser.