Rashba spin-orbit coupling in the Kane-Mele-Hubbard model