For numerous applications of quantum theory it is desirable to be able to apply arbitrary unitary operations on a given quantum system. However, in particular situations only a subset of unitary operations is easily accessible. This raises the question of what additional unitary gates should be added to a given gate set in order to attain physical universality, i.e., to be able to perform arbitrary unitary transformation on the relevant Hilbert space. In this work, we study this problem for three paradigmatic cases of naturally occurring restricted gate sets: (A) particle-number preserving bosonic linear optics, (B) particle-number preserving fermionic linear optics, and (C) general (not necessarily particle-number preserving) fermionic linear optics. Using tools from group theory and control theory, we classify, in each of these scenarios, what sets of gates are generated, if an additional gate is added to the set of allowed transformations. This allows us to solve the universality problem completely for arbitrary number of particles and for arbitrary dimensions of the single-particle Hilbert space.
