Generalized Langevin equation with a nonlinear potential of mean force and nonlinear memory friction from a hybrid projection scheme