In quantum metrology, a major application of quantum technologies, the ultimate precision of estimating an unknown parameter is often stated in terms of the Cramér-Rao bound. Yet, the latter fails to completely characterize the distribution of estimates in the nonasymptotic regime. Optimizing metrology protocols with respect to the Cramér-Rao bound can therefore lead to surprisingly poor finite-sample performance. This can be avoided by quantifying the quality of a metrology protocol by the probability of obtaining an estimate with a given accuracy. Using this intrinsically single-shot quantity naturally accommodates the finite-sample regime. We show that the fundamental limits of this figure of merit can be quantified through a multihypothesis testing problem between quantum states. Building on this connection, we derive an analogue of the Cramér-Rao bound that contains explicit corrections relevant to the finite-sample regime. We apply the finite-sample metrology framework to the example task of phase estimation with an ensemble of spin-1/2 particles, giving unambiguous evidence that large quantum Fisher information does not guarantee that a metrology protocol has a good finite-sample performance. Overall, analyzing the probability of success allows the reliable study of quantum metrology in the finite-sample regime and opens up a plethora of new avenues for research at the interface of quantum information theory and quantum metrology.
