Rocks exhibit astonishing time-dependent mechanical properties, like memory of experienced stress or slow dynamics , a transient recovery of stiffness after a softening induced by almost any type of loading. This softening and transient recovery is observed in the subsurface and in buildings after earthquakes, or in laboratory samples. Here, we investigate the anisotropy of nonlinear elastic effects in a sandstone sample under uniaxial loading. We report that slow dynamics is observed independently of propagation direction, while the acoustoelastic effect shows the expected anisotropy originating from the opening and closing of cracks. From this, we argue that slow dynamics is caused by the sliding of oblique grain-to-grain contacts and the resulting changes in frictional properties, as empirically described by rate-and-state friction and observed in laboratory experiments across block contacts. We establish a connection between the nonclassical nonlinearity of heterogeneous materials and the framework of rate-and-state friction, providing an explanation for the elusive origin of slow dynamics, and adding a different perspective for monitoring very early stages of material failure when deformation is still distributed in the bulk and begins to coalesce towards a fracture.