In the spirit of the well-known odd-number limitation, we study the failure of Pyragas control of periodic orbits and equilibria. Addressing the periodic orbits first, we derive a fundamental observation on the invariance of the geometric multiplicity of the trivial Floquet multiplier. This observation leads to a clear and unifying understanding of the odd-number limitation, both in the autonomous and the non-autonomous setting. Since the presence of the trivial Floquet multiplier governs the possibility of successful stabilization, we refer to this multiplier as the determining center. The geometric invariance of the determining center also leads to a necessary condition on the gain matrix for the control to be successful. In particular, we exclude scalar gains. The application of Pyragas control on equilibria does not only imply a geometric invariance of the determining center but surprisingly also on centers that resonate with the time delay. Consequently, we formulate odd- and any-number limitations both for real eigenvalues together with an arbitrary time delay as well as for complex conjugated eigenvalue pairs together with a resonating time delay. The very general nature of our results allows for various applications.