Memory-induced acceleration and slowdown of barrier crossing