My knowledge of Lie algebras and groups is limited, so I hope I get at least some things right.
- Are Lie algebras or groups used in cryptography?
No. Except perhaps by some daredevils, unable to communicate with the rest of us.
- Are there any, for example, cryptographic algorithms which use Lie algebras?
No. Except perhaps some super exotic ones which nobody uses.
Let's focus on the more interesting question.
- How could Lie algebras or groups be used in a cryptographic algorithm?
I'd say the two main use cases of asymmetric crypto are key exchange and signatures. Given any group, we can build both. For key exchange we use Diffie-Hellman, and for signatures we can use for example Schnorr signatures.
A Lie group has a group structure. By the above, we can use them for key exchange and signatures. The answer to the following question is crucial: how hard is the discrete logarithm problem in a Lie group? Once that is answered, the following question is crucial: what is the size of the representation of private and public keys to obtain a 128-bit security level?
Once that is answered, the following question is crucial: are we more attractive than elliptic-curve groups or multiplicative groups of finite fields?
If the answer to the last question is no, then you probably don't want to use Lie groups. My 2-cents: they seem unlikely to be useful. For the discrete logarithm problem to be hard, we'd want our group to have as little extra structure as possible. By the definition of a Lie group, we require a lot of geometric structure which has no use for our protocols. This may make the dlog problem easier, and hence parameters larger.
For Lie algebras there is no direct use-case. When I googled this, I ended up here. So apparently someone has been able to Diffie-Hellman using some Lie algebra structure. Again, ask the questions from above. If you do worse than what's already out there, what is your use-case?
Of course, perhaps there are other ways to build protocols using the Lie algebra/group structure. It could be interesting, but it may be too little too late.