To gain control over magnetic order on ultrafast time scales, a fundamental understanding of the way electron spins interact with the surrounding crystal lattice is required. However, measurement and analysis even of basic collective processes such as spin-phonon equilibration have remained challenging. Here, we directly probe the flow of energy and angular momentum in the model insulating ferrimagnet yttrium iron garnet. After ultrafast resonant lattice excitation, we observe that magnetic order reduces on distinct time scales of 1 ps and 100 ns. Temperature-dependent measurements, a spin-coupling analysis, and simulations show that the two dynamics directly reflect two stages of spin-lattice equilibration. On the 1-ps scale, spins and phonons reach quasi-equilibrium in terms of energy through phonon-induced modulation of the exchange interaction. This mechanism leads to identical demagnetization of the ferrimagnet’s two spin sublattices and to a ferrimagnetic state of increased temperature yet unchanged total magnetization. Finally, on the much slower, 100-ns scale, the excess of spin angular momentum is released to the crystal lattice, resulting in full equilibrium. Our findings are relevant for all insulating ferrimagnets and indicate that spin manipulation by phonons, including the spin Seebeck effect, can be extended to antiferromagnets and into the terahertz frequency range.