Interplay of Aharonov-Bohm and Berry phases in gate-defined graphene quantum dots