We study the sensitivity of Yu-Shiba-Rusinov states, bound states that form around magnetic scatterers in superconductors, to the presence of nonmagnetic disorder in both two and three dimensional systems. We formulate a scattering approach to this problem and reduce the effects of disorder to two contributions: disorder-induced normal reflection and a random phase of the amplitude for Andreev reflection. We find that both of these are small even for moderate amounts of disorder. In the dirty limit in which the disorder- induced mean free path is smaller than the superconducting coherence length, the variance of the energy of the Yu-Shiba-Rusinov state remains small in the ratio of the Fermi wavelength and the mean free path. This effect is more pronounced in three dimensions, where only impurities within a few Fermi wavelengths of the magnetic scatterer contribute. In two dimensions the energy variance is larger by a logarithmic factor because impurities contribute up to a distance of the order of the superconducting coherence length.