Is there still a place for obfuscation (secret algorithms) in encryption?

Question

As far as I know, the Second World War was won on codes that used obfuscation, not open source. These were also used during the cold war. For general use (not large companies with hundreds of employees who may or not be trustworthy) surely basic encryption that uses secret algorithms is still of use? I have asked this on another forum and got insulted. So if you feel the need to insult me, please do; but afterwards could you answer the question - it is genuine? A Ferrari is one of the best cars you can buy, but there is still a place for the Honda Civic. I feel the same about different encryption methods.

Answer 1

Since about 1883 when Kerckhoffs published his principles, there is no serious contest that the security of a cryptosystem should not require secrecy of it's design; that it should not be a problem if that design falls into enemy hands; that it should be that the only thing that must be kept secret for security is a (preferably short) key.

However, cryptosystems remained secret, for many reasons

1. Limitations of encryption/decryption technology (paper, mechanical device) made the above goal difficult to reach.
2. Making a cryptosystem more difficult to attack for adversaries. Unquestionably, the same algorithm kept secret is harder to attack than if it was public.
3. Preventing adversaries from using the "unbreakable" technique one has devised.
4. Preventing competitors from supplying compatible devices.
5. Ignorance.
6. Inertia.

The earliest semi-successful attempt I know to make a deliberately public (except for key) and secure cryptographic system is circa 1972, when the NBS identified the need for an unclassified algorithm. That led to DES (that was not fully successful because DES was deliberately restricted to using a key small-enough to compromise it's security).

Starting in the mid 1970's, cryptography became an academic science. Academics publish or perish, and their cryptographic designs must be public if only for this reason.

Also, publishing the details of a cryptosystem is a practical necessity for a refutation, which in turn allows quicker progress. It became a tenable position that public cryptographic designs are rather more secure (at equal key size and resource consumed) than secret ones because they benefit from the state of the art. And it's now uncontroversial that secret cryptographic algorithms are not worth academic interest (academic interest can however start when the algorithm somewhat gets published and broken, and is somewhat in use, or has been recently; e.g. Crypto 1).

And thanks to the spread of microprocessors and academic crypto, it became easy to make a practical and secure encryption system, thus reason 1 ceased to matter for users with access to technology, and reason 3 ceased to matter when it comes to even half-competent adversaries free to do as they will.

However, there are still many secret cryptographic algorithms around because

• That class includes purposely breakable puzzles for CTFs and magazines; I disregard these in the following.
• Reason 1 (public secure ciphers are complex) still applies when users wants to use only pen and paper, or must (prisoners and other people having to cope with physical searches or unavailability of technology).
• Reason 2 (secrecy helps) still holds when secrecy is paramount: the same algorithm kept secret is harder to attack than if it was public. Also, for old designs, especially with deliberately small keys, it can happen that secrecy of the design becomes part of what makes the system secure. Even in state-of-the-art cryptographic devices, secrecy of some aspects of the design (like layout of the silicon) is kept secret, for good reasons, and security evaluation even requires and quantifies that.
• Reason 3 (keep good crypto for us) did not vanish. It shifted from preventing military adversaries from using unbreakable encryption, to preventing the general public from using unbreakable encryption. A fraction of lawmakers find that worth pursuing (at least until the general public must include them), e.g. as a way to fight (add some despicable trend of a fraction of humanity). And in order for the general public to more willingfully use weak encryption, it's useful to hide evidence that the encryption is weak, and the only known way for that is keeping how it works secret. That's a strategy widely used by both dictatorships and democracies, often at several level. E.g. in GSM, A5/1 was designed to be breakable, and A5/2 more easily breakable. Thus a generation used cellphone communication intentionally made breakable. I would not bet that has changed (if only because I have no idea and like winning when I bet).
• Reason 4 (attempting to keep a juicy market captive) still applies, though it can be a mere cover for modified reason 3 above. In reality, open standard nowadays have a better chance of adoption than closed standards, thus proponents of something often make it fully open (and even more often boost that it is).
• Ignorance 5 and inertia 6 are alive and well.

---

Update: The above is about keeping an algorithm secret rather than disclosing it. As noted in comment, obfuscation now has another meaning, where one attempts to keep the algorithm or/and it's key secret despite making available a public implementation of it, or something more feasible on that tune. This is a new field, and it's not practical yet AFAIK.

Answer 2

When I served in the Israeli army we could rely on civilian encryption(AES, RSA,...) only to a limited extent. Anything serious had to be protected by Israeli military encryption. What do these magic boxes do? Well that is classified. Do they have entirely novel cryptographic primitives or do they just use a few well known algorithms? Classified of course. It is fairly easy to believe these are no weaker than Industry standards, especially since it is trivial to stack several very different ciphers so as to gurantee the result is at least as secure as either. It's a near certainty this is what is happening.

You can also add an extra secret to existing ciphers, e.g replace S boxes. And though doing this without compromising security requires a bit of knowledge it is immensely easier than building a new cipher from scratch. You get essentially a cipher with a fixed key and a variable key.

Here is a simple possible receipe for cresting your own extra secure cipher. Take a well known key stretching algorithm. Take 2 different well known ciphers. Modify one to hide a second fixed key, keep the other per standard. Take the totally novel cipher your cousin Bob invented. Stack the 3 ciphers with parts of the stretched keys one after the other. Shoot Bob.

Choosing between secret unvetted algorithm and public well vetted algorithm. That are huge advantages to the public one. But it's not an either or. You can to a great extent enjoy the best of both worlds.

I discussed military encryption with Professor Biham(Inventor of Differential cryptanalysis) He said he refused to work with Israeli army on military encryption. If he has a bright idea he wants it published, not turned into a military secret. I suspect this is common among world class cryptographers. Yet I'm also confident the military has plenty excellent cryptographers working for them.