Random unitary circuits have become a model system to investigate information scrambling in quantum systems. In the literature, mostly random circuits with Haar-distributed gate operations have been considered. In this work, we investigate operator spreading in random unitary circuits in which the elementary gate operations are drawn from general unitary-invariant ensembles, which include the well-studied Haar-distributed random unitary circuits as a special case. Similar to the Haar-distributed case, the long-time behavior of operator spreading with the more general unitary-invariant gate distribution is governed by drift-diffusion equations characterized by the butterfly velocity 𝑣B and a diffusion constant 𝒟. Differences with the Haar-random case are (i) that it takes a finite time 𝜏b until ensemble-averaged Pauli-string weights take a “binary” form, in which they depend only on whether Pauli operators inside the support of the Pauli strong are equal to the identity matrix, and (ii) that the operator spreading is characterized by a finite “domain-wall width” 𝑛DW separating regions with a random-matrix-like Pauli-string distribution. To illustrate these findings, we perform explicit calculations for random unitary circuits distributed according to the Poisson kernel, which interpolates between the trivial and Haar-distributed circuits.