We need to choose a mode of operation for a Telnet like application where the average message size is between 7 and 1024 bytes. What is the best mode of operation in this case?

a. CBC
b. CFB
c. CTR
d. Any of the above is ok

  • $\begingroup$ As the answers suggest: e. none of the above. I would go for OCB or GCM within a secure protocol $\endgroup$ – Richie Frame Apr 27 '16 at 23:55

I would pick e) none of the above. None of those modes offers integrity protection, so unless integrity is handled elsewhere, your application is wildly insecure. An attacker could modify bits in transit and do nefarious things.

Of the three, CFB and CTR are the worst for the application and should be very easy for an attacker to mount successful attacks, changing commands being sent to the server that are properly formatted. CBC would take a little more creativity, but shouldn't be too hard for an intelligent attacker.

  • $\begingroup$ Thanks for your reply. but I want to know , won't it depends on given average block size as given in question $\endgroup$ – priyanka Apr 27 '16 at 15:08
  • $\begingroup$ @priyanka, not really. For CBC, if the message is not a multiple of the block size, you can pad. Sure that adds extra bytes, but who cares these days about a couple of extra bytes. I guess if you completely ignore security and define "best" as the most compact, you could argue that a streaming mode would be "best". Ignoring security, however, is not a good idea :) $\endgroup$ – mikeazo Apr 27 '16 at 15:10

You would not just need a mode of operation for what you're asking. What you need is a secure transport protocol. Probably the best well known one for TCP connections is TLS of course. For UDP connections you could use DTLS. If you have a shared key you could use one of the pre-shared key (PSK) variants.

If you want to create your own transport protocol you could use one of the authenticated ciphers. GCM is probably the best well known one. CCM is another one that has been created specifically for packet based encryption. EAX sits somewhere in between, but isn't officially standardized by NIST. I'd highly advice against creating your own transport security though; that path is full of dangers. Just count the number of previous SSL vulnerabilities if you're unsure about that.

Transport protocols handle things like replay attacks, IV handling and the use of separate session keys where required. It could also offer perfect forward secrecy. That kind of functionality isn't provided by a mode of operation alone.

In principle you could use any of the above modes to provide confidentiality of the plaintext within transport mode security. You would however have to pair it with some function - usually a MAC or HMAC - that provides integrity and authenticity.

CBC for instance would make it somewhat harder for an attacker to change the command, but you'd immediately be vulnerable against padding oracle attacks, which completely destroy confidentiality of the message (i.e. the command and/or response).

In the unlikely case that active attacks are completely impossible you could use CFB or CTR mode as those modes can encrypt each bit separately. You'd have to make absolutely sure you never repeat the counter though.

  • $\begingroup$ Yes, please just use an off-the shelf transport protocol. Wrap your own application protocol inside of TLS or DTLS. $\endgroup$ – Stephen Touset Apr 28 '16 at 1:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.