# why a group used in cipher based on DLP must be Abelian group?

I can't understand it because $$(g^x)^y=(g^y)^x$$ in nonabelian group too.

thank you very much for read my question

• Indeed, the cyclic subgroup $\langle g\rangle$ is always abelian no matter what's happening in the ambient group. – yyyyyyy Apr 30 at 12:52

What you are asking is in some sense a tautology. In any group $$G$$, Abelian or not, the subgroup generated by an element $$g \in G$$ is Abelian. (Recall: the subgroup generated by $$g$$ is: $$\langle g\rangle := \{g^0, g^1, g^2....\}$$ ).
Hence, you can probably take your favourite DLP-based encryption scheme and implement it in $$G$$, without dealing with the fact that it is not Abelian. Indeed, the only fact you need is the existence of a subgroup $$G_1$$ with large prime order $$p$$.
For example, consider a large prime $$p$$ and the corresponding Dihedral group $$D_p$$. It is one of the simplest examples for a non-Abelian group, and it has a subgroup of order $$p$$. Let $$g$$ be random element of order $$p$$ (that is easy to verify, since elements in $$D_p$$ are either of order $$p$$ or 2). Now, you define El-Gamal encryption as usual:
Secret key is $$x \in \mathbb{Z}_p^*$$ to your choice, public key is $$(D_p, p, g, h = g^x)$$. We encrypt a message $$m \in D_p$$ by choosing a random $$y\in \mathbb{Z}_p^*$$ and computing $$m \mapsto (c_1, c_2) = (g^y, m \cdot h^y),$$ and decrypt by computing $$(c_1, c_2) \mapsto c_2 \cdot c_1^{-x}.$$
Note that in an Abelian group, we could also decrypt by multiplying in a different order: $$(c_1, c_2) \mapsto c_1^{-x} \cdot c_2,$$ but, well, who cares - just remember to multiply in the right order.
An interesting question in my opinion, is what can be gained by considering an underlying group that is not Abelian. Indeed, if the presentation of the group if complicated, perhaps DLP is harder. This is the case with elliptic curve based cryptography: the groups considered are cyclic as usual, but the presentation of the group is far more complicated than $$\mathbb{Z}_p$$, leading to better security against DLP.