How effective is Serpent-256 in CTR mode?

Question

I downloaded a compression program known as PeaZip which offers encryption and 2-way authentication also.

PeaZip has an option to encrypt compressed file with variety of ciphers and one of them is Serpent.

It gives option of:

1. Serpent 128 CTR OMAC
2. Serpent 256 CTR OMAC

I want to know how strong any cipher, specifically Serpent, is in CTR mode. Assuming a strong password or truly random key file of appropriate size is used.

I am not aware of how effectively this cipher is handled in PeaZip, but that is a different matter.

Ideally, in which mode Serpent is more effective.

-- Edited --

The details of crypto library which PeaZip uses are here.

Author of this library says about Serpent:

Serpent (designed by Ross Anderson, Eli Biham, and Lars Knudsen) was one of the AES finalists; it is a 128-bit block cipher with key sizes of 128, 192, or 256 bits. Please note that although my code supports OMAC and EAX, these modes are not tested against public test vectors (the usual C libraries do either not implement Serpent or OMAC/EAX).

Answer 1

I want to know how strong any cipher, specifically Serpent, is in CTR mode.

CTR mode has a security proof. When you use it right (no nonce collisions, etc.) and with a secure block cipher (full Serpent has not been attacked), it will give you confidentiality.

OMAC is a MAC algorithm, or a pair of them actually, which is meant to authenticate the message, proving that it has not been tampered with. It also has a security proof.

So correctly used those are fine. One potential issue is whether and how the key is reused for them, however. EAX (pdf) is an authenticated encryption mode with CTR+OMAC1 that does make sure to use them correctly from that point of view. I have no idea what PeaZip does.

Ideally, in which mode Serpent is more effective.

It does not really matter. Any secure block cipher can be used in any secure mode. Optimally you want an authenticated encryption mode like GCM, or a secure composition of encryption and authentication modes.

The only common modes I would recommend staying clear of are ECB (almost never the right choice) and CBC (theoretically secure, but practical implementations have been weak). CTR is good as long as you authenticate it, as PeaZip seems to do.

Answer 2

Any block cipher within CTR operates the same way; CTR security is only dependent on the block size. Serpent was one of the candidates to become AES. It is thought to have a high security margin (and no weak keys etc). None of the currently known attacks seem to have done much damage to the security claims.

All AES candidates have a block size of 128 bit and a key size of 128, 192 and 256 bits (at least). In your case you just list two key sizes, 128 bit or 256 bit as difference. The higher key size is more secure. As the number of rounds of Serpent doesn't change the greater sized key should be preferred, although the strength of the password is much more important.

In the end, an algorithm is effective when it handles your specific requirements. And in general, the security of the system is very likely not much influenced by the choice of Serpent as block cipher (especially if timing attacks are hard to do such as with file encryption, hardware implementations of AES such as AES-NI may have a small advantage with regards to timing based attacks, besides the obvious performance benefit).