Dynamical backaction cooling has been demonstrated to be a successful method for achieving the motional quantum ground state of a mechanical oscillator in the resolved-sideband regime, where the mechanical frequency is significantly larger than the cavity decay rate. Nevertheless, as mechanical systems increase in size, their frequencies naturally decrease, thus bringing them into the unresolved-sideband regime, where the effectiveness of the sideband cooling approach decreases. Here we demonstrate, however, that this cooling technique in the unresolved-sideband regime can be significantly enhanced by utilizing a nonlinear cavity as shown in the experimental work of Zoepfl et al. [Phys. Rev. Lett. 130, 033601 (2023)]. The above arises due to the increased asymmetry between the cooling and heating processes, thereby improving the cooling efficiency. In addition, we show that injecting a squeezed vacuum into the nonlinear cavity paves the way to ground-state cooling of the mechanical mode. Notably, the required squeezing parameters are far less stringent than in the linear case, simplifying experimental implementation.
