What can unitary sequences tell us about multi-time physics?