We study the influence of a magnetic flux tube on the possibility to electrostatically confine electrons in a graphene quantum dot. Without a magnetic flux tube, the graphene pseudospin is responsible for a quantization of the total angular momentum to half-integer values. On the other hand, with a flux tube containing half a flux quantum, the Aharonov-Bohm phase and Berry phase precisely cancel, and we find a state at zero angular momentum that cannot be confined electrostatically. In this case, true bound states only exist in regular geometries for which states without zero-angular-momentum component exist, while nonintegrable geometries lack confinement. We support these arguments with a calculation of the two-terminal conductance of a gate- defined graphene quantum dot, which shows resonances for a disk-shaped geometry and for a stadium-shaped geometry without flux tube, but no resonances for a stadium-shaped quantum dot with a π-flux tube.