Serpent vs AES,, theoretical security

In the lets-choose-the-next-aes contest in 1998, they chose Rijndael because of speed. Now, not taking speed into account, is Serpent more secure or not? Since it has 32 rounds (of course that's not that useful if the round algorithm isn't secure,) I thought it's more secure, since it has the same block size, key sizes AND more rounds. But I really don't know the details, so if you threw away efficiency on x86 CPUs and ease of implementation, then Serpent would be more secure, right? If yes, is it superior, or just a tiny bit more secure?

The number of rounds does not directly influence security; it depends at least as much on the complexity of those rounds. That said, it is possible to use the percentage of rounds for which there are attacks as an indication on how much crypt-analysis has advanced (disregarding those attacks that are worse than brute force, of course).

One tricky thing with this is memory usage; many attacks are still better than the time-complexity of brute force, but have impossible memory requirements. I don't think there is any objective way to weigh memory-complexity and time-complexity.

Another thing to notice is that these attacks are far from practical; you can possibly see them as indication if other / more advanced attacks are possible. Of course, if this is true cannot be predicted, but for sure some attacks are more likely to be "extended" than others as well.

Still, with that said, for Serpent-128 it seems that 11 out of 32 rounds have been shown to be weak in an attack. For AES-128 there is the biclique attack on the full cipher. This attack however only brings down the attack complexity to $$2^{126.1}$$; having a complex attack that has a smaller than 4 times advantage over a brute force attacks doesn't say much. Otherwise, there are attacks for at least 7 rounds out of the 10 used for AES-128 (the other key sizes use a larger number of rounds).

So in that sense Serpent should be considered to be much more secure than AES; $$\frac{11}{32} \ll \frac{7}{11}$$ after all. AES is however also more than twice as efficient. In that sense Serpent with 16 rather than 32 rounds would probably have fared better at the competition. However, that's just for the minimum key sizes. If you look at AES-256 then you would have all the security you may need (9 out of 14 rounds or $$2^{254.3}$$ for the biclique attack) - and still be faster than even Serpent-128. And that's without hardware acceleration. Note that I've disregarded related key attacks on AES-256 in this scenario.

The notion on Serpent that "It is build like a tank" (Cryptography Engineering: Design Principles and Practical Applications) still seems to hold. Note that there haven't been many breakthroughs on either cipher in the last couple of years; research seems to have moved on.

• I think a good example of more rounds doesn't imply better security would be Threefish because it has 72 but they're all extremely simple (think like 1 add, 1 xor and 1 rotate on each word). – SEJPM Dec 8 '19 at 19:07