A major challenge for underwater passive detection in shallow-water waveguides is the accurate separation of normal modes from passive target-radiated signals (such as ship-radiated noise and opportunistic source signals) using a horizontal line array. Existing methods typically handle each frequency independently, so they fail to fully utilize the correlation of modes across frequencies. As a result, stringent requirements are imposed on array aperture and signal-to-noise ratio (SNR) to achieve reliable modal separation. To address this issue, the waveguide invariant is incorporated as a physical prior, and a cross-frequency-consistent dictionary of modal horizontal wavenumbers is constructed under the constraint of the shallow-water modal dispersion relation. Based on this dictionary, multi-frequency jointly sparse modal separation is carried out within a sparse Bayesian learning framework. Numerical simulations in a benchmark shallow-water environment show that compared with representative existing methods, the proposed method achieves higher modal separation accuracy and reduces the array aperture required to separate all propagation modes by more than 20%, while maintaining high separation accuracy and robustness under low-SNR conditions. Its performance is further improved as the frequency spacing decreases. Moreover, this method benefits from a more precise waveguide invariant and is more sensitive to underestimation than overestimation. The low-order modes can still be separated with reasonably high accuracy even when there is a significant mismatch in the waveguide invariant. Finally, the proposed method is validated using passive experimental data collected from a horizontal line array deployed in a shallow-water region of the South China Sea in 2021. The sea-trial results further demonstrate its feasibility in realistic ocean environments.