We need to choose a mode of operation for a Telnet like application where the average message size is between 7 and 1024 bytes. What is the best mode of operation in this case?
d. Any of the above is ok
Cryptography Stack Exchange is a question and answer site for software developers, mathematicians and others interested in cryptography. It only takes a minute to sign up.Sign up to join this community
I would pick e) none of the above. None of those modes offers integrity protection, so unless integrity is handled elsewhere, your application is wildly insecure. An attacker could modify bits in transit and do nefarious things.
Of the three, CFB and CTR are the worst for the application and should be very easy for an attacker to mount successful attacks, changing commands being sent to the server that are properly formatted. CBC would take a little more creativity, but shouldn't be too hard for an intelligent attacker.
You would not just need a mode of operation for what you're asking. What you need is a secure transport protocol. Probably the best well known one for TCP connections is TLS of course. For UDP connections you could use DTLS. If you have a shared key you could use one of the pre-shared key (PSK) variants.
If you want to create your own transport protocol you could use one of the authenticated ciphers. GCM is probably the best well known one. CCM is another one that has been created specifically for packet based encryption. EAX sits somewhere in between, but isn't officially standardized by NIST. I'd highly advice against creating your own transport security though; that path is full of dangers. Just count the number of previous SSL vulnerabilities if you're unsure about that.
Transport protocols handle things like replay attacks, IV handling and the use of separate session keys where required. It could also offer perfect forward secrecy. That kind of functionality isn't provided by a mode of operation alone.
In principle you could use any of the above modes to provide confidentiality of the plaintext within transport mode security. You would however have to pair it with some function - usually a MAC or HMAC - that provides integrity and authenticity.
CBC for instance would make it somewhat harder for an attacker to change the command, but you'd immediately be vulnerable against padding oracle attacks, which completely destroy confidentiality of the message (i.e. the command and/or response).
In the unlikely case that active attacks are completely impossible you could use CFB or CTR mode as those modes can encrypt each bit separately. You'd have to make absolutely sure you never repeat the counter though.