Suppression of hole decoherence in ultrafast photoionization