In the field of quantum magnetism, the advent of numerous spin-orbit assisted Mott insulating compounds, such as the family of Kitaev materials, has led to a growing interest in studying general spin models with nondiagonal interactions that do not retain the SU(2) invariance of the underlying spin degrees of freedom. However, the exchange frustration arising from these nondiagonal and often bond-directional interactions for two- and three-dimensional lattice geometries poses a serious challenge for numerical many-body simulation techniques. In this paper, we present an extended formulation of the pseudofermion functional renormalization group that is capable of capturing the physics of frustrated quantum magnets with generic (diagonal and off-diagonal) two-spin interaction terms. Based on a careful symmetry analysis of the underlying flow equations, we reveal that the computational complexity grows only moderately, as compared to models with only diagonal interaction terms. We apply the formalism to a kagome antiferromagnet which is augmented by general in-plane and out-of-plane Dzyaloshinskii-Moriya (DM) interactions, as argued to be present in the spin liquid candidate material herbertsmithite. We calculate the complete ground state phase diagram in the strength of in-plane and out-of-plane DM couplings, and discuss the extended stability of the spin liquid of the unperturbed kagome antiferromagnet in the presence of these couplings.
