Improved spectral gaps for random quantum circuits: Large local dimensions and all-to-all interactions