Recovering Quantum Gates from Few Average Gate Fidelities