Cryptography Stack Exchange is a question and answer site for software developers, mathematicians and others interested in cryptography. It only takes a minute to sign up.
Sign up to join this community
Proprietary software generally relies on the fact that in keeping the encryption algorithm private, it gets an extra layer of security implying "Security through Obscurity." Obviously this phrase has been hotly debated and surely is only acceptable if discovering the mechanics of a proprietary algorithm doesn't actually reduce it's security.
Is there any other reason you would use a proprietary encryption since my research has proven there to be many more disadvantages than advantages!
Yes, there are advantages to the attacker.
Using a well vetted encryption algorithm provides a better assurance of security. There may be cryptographic algorithm flaws and/or coding mistakes.
As noted, relying on the algorithm being private just adds a layer of false security.
The main advantage is that using a proprietary algorithm gives you access to trade secrets like additional cryptographic attacks that other algorithms fall to but to which the proprietary algorithm is resistant.
Whether this is important depends on the amount of trust you have in the vendor. As other answers have noted, usually the staff of any one organization is too small and underfunded to do find as many problems as might be found by the open peer-review process, so usually proprietary algorithms are a bad idea.
However, certain large groups (e.g. large governments and their militaries) have important trade secrets and enormous staffs containing some of the best cryptographers in the world. If I were sitting in a United States government office, doing government business that the NSA wanted to keep secret and the head of cryptography in the NSA gave me a black-box to encrypt my communications, I'd probably use it over using a more public system. $10+ billion/year buys a lot of peers to review things and we know they discovered differential cryptanalysis before IBM discovered it in 1974 (and then negotiated to keep it secret). It was almost 20 years before the public was aware of this type of attack.
The only advantage I can think of is that they're able to put "State of the art encryption" on their website. But even then, those with a trained eye may spot it as an issue, therefore rendering it as yet another disadvantage.
But other than that pseudo-advantage, there are none.
Custom crypto can be valuable when other aspects are more important than the confidentiality guarantee, and the well-known ciphers don't address those aspects.
A custom cipher or custom application of a cipher would tend to offer a weaker guarantee of confidentiality than well-tested systems. But some users of encryption can handle an eventual breach so long as they can successfully sue the culprit. This is the case for Digital Restrictions Management (DRM), for example. The Content Scrambling System (CSS) in DVD-Video used a custom cipher that ended up being trivial to break for two reasons: a known-plaintext attack against predictable parts of the stream, and the fact that it was limited to 40 bits to satisfy U.S. munitions export regulations applicable in the mid-1990s. But it was established in Universal v. Reimerdes that the owner of copyright in a movie distributed with a particular DRM scheme has a legal case against those who would break the scheme. Just being crypto was found to be enough under the law.
Another application of custom crypto might be for games running on an 8-bit or 16-bit microprocessor. Several games for third- and fourth-generation video game consoles (1985 through 1996) offer a password system to let the user save the progress of his campaign by writing it down and then keying it in later. Because keying in a password was so laborious, the messages had to be kept short: 32 bits of data plus 8-bit check for an 8-character password or 72 bits of data plus 8-bit check for a 16-character password. This was often smaller than the block size of the ciphers that were published at the time, such as DES, and DES also took quite a bit of ROM space to implement, ROM space that could better be spent on, say, adding more levels to a game. Besides, the crypto had to be secure only against casual black-box analysis, as tools for "ROM dumping" weren't widespread until the fourth generation ended.
A possible advantage is the need for cryptanalysis.
Using only standard algorithms, an adversary who had a machine capable of breaking them could just feed your ciphertext into the machine. With a proprietary algorithm they would not have a ready-made machine for breaking it, so they would have to analyze it first, even if after that it would be very easy to break.
For this to be a benefit, you first have to assume that standard algorithms are breakable. Second, you cannot use your algorithm for long or the adversary could have time for cryptanalysis and create a machine for breaking it quickly.
Needless to say, I do not think this is a normally realistic scenario. However, in e.g. a time of war with an adversary whose capabilities you do not know, using obscurity as an additional layer of defense on top of standard algorithms might be justified.
Generally speaking, using proprietary encryption is a major problem, because the algorithms which are used are not subject to the same amount of review they would be if public. But it is possible to gain some advantages this way, by taking advantage of public knowledge.
Maurer & Massey (1993) prove that a cascade of ciphers with independent keys is at least as strong as its first cipher. So while PROPRIETARY(x) might not protect data as well as AES(x), and even AES(PROPRIETARY(x)) could be worse, PROPRIETARY(AES(x)) is at least as strong as AES(x).
For example, suppose PROPRIETARY(x) takes a key of length log_2((#x)!) bits and shuffles the bits according to the key. This is a weak cipher, subject (for example) to a known-plaintext attack. It seems unlikely that AES(PROPRIETARY(x)) would be weak, but it can be proven that PROPRIETARY(AES(x)) is at least as strong as AES itself.
Of course nothing in this post should be taken as a reason to use proprietary encryption. In addition to taking resources to create and especially maintain, they can cause a false sense of security.
I can think of only one answer: DRM.
Or rather, anywhere you have a system in which the consumer must be able to decrypt and consume the encrypted data in a controlled fashion, but at the same time must not be able to do so freely.
Consider, as an example, a 2D game with user-provided content, where users can create graphical assets for their own areas, but don't want other players to be able to steal those assets for their areas.
Now, this requirement is physically impossible for any large volume of user-created assets - the data must be decrypted to send it to the videocard and display it. But people dislike having their work ripped off. The obvious primary approach is to take a social approach to it: make it just plain uncool to rip off people's work, name and shame, assist 3rd-party toolmakers who play nice, censure those who don't, provide in-game admin policing of reported copyvios, assist in legal disputes... but that's all super-high-overhead, taking immense amounts of admin time, so there have to be technical methods to help.
And you can only get so far with automated dup-checking of uploads, using custom filetypes with custom editors which track attribution history, and so forth.
So the hurdle is to make it harder to steal a graphic by decoding it, than it is to steal it by other possible methods (screenshotting, etc).
All you're doing there is scrambling, you're not encrypting, because you can't legitimately call something encryption if you're handing the key to every possible recipient ever.
And the end result is that you end up with an arms race, where you permutate, tweak and improve your scrambling a little each release. You can't even make it the very best scrambling that you could, because then once they cracked that, you'd not have anything further to move to in the arms race.
Edit 1: So why not use an open algorithm? Well, with DRM, your only defense against cracking becomes security through obscurity. Using standard crypto is a significant disadvantage here, as it is explicitly designed to be as non-obscure as possible. It will be the first thing people try; there are plentiful libraries for them to use to make code to attack your data; and there is plentiful documentation and people with skill in using them to answer any questions the hackers might have.
Edit 2: The question description says: "Obviously this phrase has been hotly debated and surely is only acceptable if discovering the mechanics of a proprietary algorithm doesn't actually reduce it's security."
I do not believe that this argument holds water, in a logical sense, either for DRM or for real encryption.
If revealing an algorithm doesn't reduce its security, then there is no reason not to reveal it.
There is also every reason TO reveal it. Revealing has advantages, while concealing it has none. A concealed algorithm has no trust. If you reveal, you can also patent it, you are then protected from theft far better than just hoping that someone doesn't reverse compile it. You can also bask in the attention and adulation of the crowds for creating an impenetrable algorithm that was better than existing open solutions.
Concealment just suggests that you have something to hide: that you feel you need the security through obscurity. Only DRM needs that, and that's only because "DRM encryption" is an oxymoron.
Keeping the Algorithm secure gives an advantage. Like GOST cipher was kept secret until its public release in 1994 whereas it was being used by Russians in 1970 too.
So what we learn is, Algorithm must be vetted for security by experts and then efforts should be made to keep it secret from adversary. But just trying to keep a weak cipher secret only provides false security because one never knows how and when information has been compromised.
Obscurity can work if your application is, indeed, obscure. If you're Microsoft, that won't work. If millions of dollars depend on the security, that won't work either.
But don't make up your own encryption algorithm - chances are it'll be easy to break. Instead, modify an existing encryption algorithm that's already good. For instance, use AES but swap the 2nd and 7th bytes in each 16-bit block. Or swap the 2nd and 7th byte in the KEY. Then, you've got the security of AES, but it's not off-the-shelf AES.
I think the answer is in some ways very obvious and in some ways less obvious.
The advantage is that if it's proprietary, then that means you're almost certainly paying another party for it and therefore holding them responsible for reviewing, maintaining, and debugging the encryption library when problems arise.
Contrast this with a library like OpenSSL, which turned out to have serious hidden flaws for prolonged periods of time due to the lack of sufficient manpower and funding for its maintenance and review.
(Note: I'm merely mentioning the advantages, not claiming that they outweigh the disadvantages or that there are no better alternatives.)
