We investigate the quantum phases of the frustrated spin-12J1−J2−J3 Heisenberg model on the square lattice with ferromagnetic J1 and antiferromagnetic J2 and J3 interactions. Using the pseudofermion functional renormalization group technique, we find an intermediate paramagnetic phase located between classically ordered ferromagnetic, stripy antiferromagnetic, and incommensurate spiral phases. We observe that quantum fluctuations lead to significant shifts of the spiral pitch angles compared to the classical limit. By computing the response of the system with respect to various spin rotation and lattice symmetry-breaking perturbations, we identify a complex interplay between different nematic spin states in the paramagnetic phase. While retaining time-reversal invariance, these phases either break spin-rotation symmetry, lattice-rotation symmetry, or a combination of both. We therefore propose the J1−J2−J3 Heisenberg model on the square lattice as a paradigmatic example where different intimately connected types of nematic orders emerge in the same model.
