The key point in RSA is the fact that inversion in modulo $|G|$ is hard, of course other groups are a priorpriori okay to build a secure encryption scheme.
For example U_{pqr}$U_{pqr}$, with $p, q, r$ large primes seems to be okay. But it will be probably less efficient (keys and ciphertexts will be probably larger for the same level of security).
Of course, it is not because the problem seems to be hard, that we can conclude with the same level of confidency that the RSA-problem-hardness. The RSA-problem has been particularly well-studied. And if you choose to change the group used, the risk to discover that "inversion in modulo $|G|$ is not so hard" is more important.