# How effective is Serpent-256 in CTR mode?

I downloaded a compression program known as PeaZip which offers encryption and 2-way authentication also.

PeaZip has an option to encrypt compressed file with variety of ciphers and one of them is Serpent.

It gives option of:

1. Serpent 128 CTR OMAC
2. Serpent 256 CTR OMAC

I want to know how strong any cipher, specifically Serpent, is in CTR mode. Assuming a strong password or truly random key file of appropriate size is used.

I am not aware of how effectively this cipher is handled in PeaZip, but that is a different matter.

Ideally, in which mode Serpent is more effective.

-- Edited --

The details of crypto library which PeaZip uses are here.

Author of this library says about Serpent:

Serpent (designed by Ross Anderson, Eli Biham, and Lars Knudsen) was one of the AES finalists; it is a 128-bit block cipher with key sizes of 128, 192, or 256 bits. Please note that although my code supports OMAC and EAX, these modes are not tested against public test vectors (the usual C libraries do either not implement Serpent or OMAC/EAX).

• By "which mode" do you mean the two options you list or in general? – otus Sep 2 '16 at 8:01
• @otus I mean modes in general. – RPK Sep 2 '16 at 12:59
• That's a bit of a strange excuse. You can test EAX with AES as the test vectors are in the paper, appendix G. Furthermore, I'm sure there are Serpent test vectors out there. OK, possibly there are no test vectors for the combination, but AES has the exact same interface (of course, as Serpent was an AES candidate). – Maarten Bodewes Sep 2 '16 at 14:47

I want to know how strong any cipher, specifically Serpent, is in CTR mode.

CTR mode has a security proof. When you use it right (no nonce collisions, etc.) and with a secure block cipher (full Serpent has not been attacked), it will give you confidentiality.

OMAC is a MAC algorithm, or a pair of them actually, which is meant to authenticate the message, proving that it has not been tampered with. It also has a security proof.

So correctly used those are fine. One potential issue is whether and how the key is reused for them, however. EAX (pdf) is an authenticated encryption mode with CTR+OMAC1 that does make sure to use them correctly from that point of view. I have no idea what PeaZip does.

Ideally, in which mode Serpent is more effective.

It does not really matter. Any secure block cipher can be used in any secure mode. Optimally you want an authenticated encryption mode like GCM, or a secure composition of encryption and authentication modes.

The only common modes I would recommend staying clear of are ECB (almost never the right choice) and CBC (theoretically secure, but practical implementations have been weak). CTR is good as long as you authenticate it, as PeaZip seems to do.

• I think you are either confusing CCM with EAX or CBC-MAC with OMAC. CCM uses CBC-MAC (which is insecure if the length isn't known) and EAX uses OMAC/CMAC. CCM is mainly used for packet encryption, so in this case EAX would make (slightly) more sense. – Maarten Bodewes Sep 2 '16 at 13:11
• @MaartenBodewes, indeed, I was thinking about EAX. Thanks, fixed. – otus Sep 2 '16 at 13:18
• @otus Please see edited original post. – RPK Sep 2 '16 at 14:26
• @RPK, so looks like EAX, which is fine. – otus Sep 2 '16 at 14:28

Any block cipher within CTR operates the same way; CTR security is only dependent on the block size. Serpent was one of the candidates to become AES. It is thought to have a high security margin (and no weak keys etc). None of the currently known attacks seem to have done much damage to the security claims.

All AES candidates have a block size of 128 bit and a key size of 128, 192 and 256 bits (at least). In your case you just list two key sizes, 128 bit or 256 bit as difference. The higher key size is more secure. As the number of rounds of Serpent doesn't change the greater sized key should be preferred, although the strength of the password is much more important.

In the end, an algorithm is effective when it handles your specific requirements. And in general, the security of the system is very likely not much influenced by the choice of Serpent as block cipher (especially if timing attacks are hard to do such as with file encryption, hardware implementations of AES such as AES-NI may have a small advantage with regards to timing based attacks, besides the obvious performance benefit).

• As the number of rounds of Serpent doesn't seem to change; it doesn't; in fact, given a 128 bit key, Serpent does a simple mapping to a 256 bit key (setting the missing 128 bits to a 1000...000 pattern), and then uses the 256 bit algorithm. The Serpent designers have stated that they did this specifically so they don't people an incentive for using 128 bit keys. – poncho Sep 2 '16 at 13:11
• Edited original post. – RPK Sep 2 '16 at 14:25
• @RPK Doesn't matter much. Note that EAX is constructed from CTR and OMAC mode, so using either one of them should be secure. EAX leaves less room for mistakes as it e.g. already includes the IV into the calculation. Neither of them responds well if the IV (or nonce really) repeats. – Maarten Bodewes Sep 2 '16 at 14:43