Are there any advantages in using proprietary encryption?

Proprietary software generally relies on the fact that in keeping the encryption algorithm private, it gets an extra layer of security implying "Security through Obscurity." Obviously this phrase has been hotly debated and surely is only acceptable if discovering the mechanics of a proprietary algorithm doesn't actually reduce it's security.

Is there any other reason you would use a proprietary encryption since my research has proven there to be many more disadvantages than advantages!

-
I don't think it's much of a debate that "security through obscurity" is a baaaad idea. – Daffy Jan 15 at 16:31
There is also Schneier's Law: "Anyone, from the most clueless amateur to the best cryptographer, can create an algorithm that he himself can't break." – zaph Jan 15 at 16:35
This has been covered in incredible depth over at Security. SE. See security.stackexchange.com/q/66552/971, security.stackexchange.com/q/24449/971, security.stackexchange.com/q/44094/971, security.stackexchange.com/q/2430/971, security.stackexchange.com/q/32064/971, security.stackexchange.com/a/2210/971. I'm not sure there's much purpose in repeating all of those points here... – D.W. Jan 15 at 20:36
@user1193112 On the contrary. There are many who believe "security through obscurity" is bad (mostly because they have read this phrase everywhere). It's more correct to say that it's bad in the absence of real security. When added to real security, it does offer some additional benefit. For example, locking your session when you step away from your computer is security through obscurity. But it will still stop a casual nosy person from snooping around while you're in the bathroom. – JBentley Jan 15 at 20:36
Locking your screen is not security through obscurity. Obscurity means that the code / algorithm etc. is obscure (i.e., hidden inside some TPM module, only available in binary executable, or closed away in a wooden box with some handles on the outside). The screen saver is locked by a password, which is perfectly un-obscure. – AnoE Jan 15 at 21:29

Yes, there are advantages to the attacker.

Using a well vetted encryption algorithm provides a better assurance of security. There may be cryptographic algorithm flaws and/or coding mistakes.

As noted, relying on the algorithm being private just adds a layer of false security.

-
+1 for "Yes, there are advantages to the attacker." – Daffy Jan 15 at 16:31
there are plenty of flaws and coding mistakes in extremely well vetted pieces of software that use open algorithms, being proprietary does not necessarily make that more likely – Richie Frame Jan 16 at 1:50
It is true that open software is not necessarily more free of bugs but algorithms such as AES which were exceptionally well vetted through a rigorous completion and even after well studied should be substantially better vetted than a proprietary algorithm that has not have nearly the scrutiny. Further implementations that are FIPS certified have had their code critically examined. It is true that well funded governments, for example, have the facilities, time and talent to produce proprietary algorithms that rival or exceed standards but are the exception an not under consideration here. – zaph Jan 16 at 3:08
... more: There is of course another threat to standard algorithm, that of a government tampering with the them. The NSA proposed changes to the S-boxes of DES, it is not know why, no explanation was given. Then there is the Dual_EC_DRBG controversy. – zaph Jan 16 at 3:11
the nsa tampered with the s-boxes to make the algorithm stronger, quite a bit so – Richie Frame Jan 16 at 11:28

The main advantage is that using a proprietary algorithm gives you access to trade secrets like additional cryptographic attacks that other algorithms fall to but to which the proprietary algorithm is resistant.

Whether this is important depends on the amount of trust you have in the vendor. As other answers have noted, usually the staff of any one organization is too small and underfunded to do find as many problems as might be found by the open peer-review process, so usually proprietary algorithms are a bad idea.

However, certain large groups (e.g. large governments and their militaries) have important trade secrets and enormous staffs containing some of the best cryptographers in the world. If I were sitting in a United States government office, doing government business that the NSA wanted to keep secret and the head of cryptography in the NSA gave me a black-box to encrypt my communications, I'd probably use it over using a more public system. $10+ billion/year buys a lot of peers to review things and we know they discovered differential cryptanalysis before IBM discovered it in 1974 (and then negotiated to keep it secret). It was almost 20 years before the public was aware of this type of attack. - It's worth noting though that back in 1974 there wasn't much interesting from mathematicians in protecting things, you can write letters in the classic book cyphers that are still impractical to break. I am not sure that this is relevant. – Alec Teal Jan 19 at 16:47 The only advantage I can think of is that they're able to put "State of the art encryption" on their website. But even then, those with a trained eye may spot it as an issue, therefore rendering it as yet another disadvantage. But other than that pseudo-advantage, there are none. 1. Chances are overwhelmingly good that this new cipher, having been constructed in secret, where peer review is impossible, will have many flaws. 2. Even if the original implementation of the cipher is kept secret, it can almost certainly be reverse engineered. 3. The newly made primitives won't be as tested and publicly reviewed as more common ciphers like AES or DES (which, by the way, started its life as proprietary, created by IBM in secret. We all know how DES turned out) 4. It takes a lot of time and money to get professionals who will design the primitives for you. 5. Worse still, if you can't afford the time or money it takes to get professionals, then there won't be nearly as much skill behind this new cipher, making all the previous issues worse. - @zaph By that, I meant most ciphers that have been created previously have been discovered to have an issue that brings their effective security down to a brute-forcable range. – Daffy Jan 15 at 16:58 @zaph I see your point, and I've edited it. In hindsight, I agree with you that it was too broad. – Daffy Jan 15 at 17:22 The relevelent question is about current "best practice" ciphers such as AES and has "their effective security down to a brute-forcable range". The edited question does not address the comment which still seems to be more FUD that useful. – zaph Jan 15 at 17:40 I wouldn't call it "state of the art" to use an proprietary (i.e. not widely analyzed) algorithm. Maybe "cutting edge" is more correct (and even sounds cooler). – Paŭlo Ebermann Jan 17 at 17:46 @Daffy "We all know how DES turned out" You mean like how it was the workhorse of symmetric encryption for something like two decades, and that even now, close to 40 years after it was standardized, the very best attack (says Wikipedia, at least) on its 56-bit key needs 2^43 known plaintexts (about 7e13 bytes = 70 TB encrypted under the same key) and around 2^40 time? I'd call that a pretty good track record! Certainly not very good as a cautionary tale. – Michael Kjörling Jun 19 at 16:29 Custom crypto can be valuable when other aspects are more important than the confidentiality guarantee, and the well-known ciphers don't address those aspects. A custom cipher or custom application of a cipher would tend to offer a weaker guarantee of confidentiality than well-tested systems. But some users of encryption can handle an eventual breach so long as they can successfully sue the culprit. This is the case for Digital Restrictions Management (DRM), for example. The Content Scrambling System (CSS) in DVD-Video used a custom cipher that ended up being trivial to break for two reasons: a known-plaintext attack against predictable parts of the stream, and the fact that it was limited to 40 bits to satisfy U.S. munitions export regulations applicable in the mid-1990s. But it was established in Universal v. Reimerdes that the owner of copyright in a movie distributed with a particular DRM scheme has a legal case against those who would break the scheme. Just being crypto was found to be enough under the law. Another application of custom crypto might be for games running on an 8-bit or 16-bit microprocessor. Several games for third- and fourth-generation video game consoles (1985 through 1996) offer a password system to let the user save the progress of his campaign by writing it down and then keying it in later. Because keying in a password was so laborious, the messages had to be kept short: 32 bits of data plus 8-bit check for an 8-character password or 72 bits of data plus 8-bit check for a 16-character password. This was often smaller than the block size of the ciphers that were published at the time, such as DES, and DES also took quite a bit of ROM space to implement, ROM space that could better be spent on, say, adding more levels to a game. Besides, the crypto had to be secure only against casual black-box analysis, as tools for "ROM dumping" weren't widespread until the fourth generation ended. - A possible advantage is the need for cryptanalysis. Using only standard algorithms, an adversary who had a machine capable of breaking them could just feed your ciphertext into the machine. With a proprietary algorithm they would not have a ready-made machine for breaking it, so they would have to analyze it first, even if after that it would be very easy to break. For this to be a benefit, you first have to assume that standard algorithms are breakable. Second, you cannot use your algorithm for long or the adversary could have time for cryptanalysis and create a machine for breaking it quickly. Needless to say, I do not think this is a normally realistic scenario. However, in e.g. a time of war with an adversary whose capabilities you do not know, using obscurity as an additional layer of defense on top of standard algorithms might be justified. - Generally speaking, using proprietary encryption is a major problem, because the algorithms which are used are not subject to the same amount of review they would be if public. But it is possible to gain some advantages this way, by taking advantage of public knowledge. Maurer & Massey (1993) prove that a cascade of ciphers with independent keys is at least as strong as its first cipher. So while PROPRIETARY(x) might not protect data as well as AES(x), and even AES(PROPRIETARY(x)) could be worse, PROPRIETARY(AES(x)) is at least as strong as AES(x). For example, suppose PROPRIETARY(x) takes a key of length log_2((#x)!) bits and shuffles the bits according to the key. This is a weak cipher, subject (for example) to a known-plaintext attack. It seems unlikely that AES(PROPRIETARY(x)) would be weak, but it can be proven that PROPRIETARY(AES(x)) is at least as strong as AES itself. Of course nothing in this post should be taken as a reason to use proprietary encryption. In addition to taking resources to create and especially maintain, they can cause a false sense of security. - I can think of only one answer: DRM. Or rather, anywhere you have a system in which the consumer must be able to decrypt and consume the encrypted data in a controlled fashion, but at the same time must not be able to do so freely. Consider, as an example, a 2D game with user-provided content, where users can create graphical assets for their own areas, but don't want other players to be able to steal those assets for their areas. Now, this requirement is physically impossible for any large volume of user-created assets - the data must be decrypted to send it to the videocard and display it. But people dislike having their work ripped off. The obvious primary approach is to take a social approach to it: make it just plain uncool to rip off people's work, name and shame, assist 3rd-party toolmakers who play nice, censure those who don't, provide in-game admin policing of reported copyvios, assist in legal disputes... but that's all super-high-overhead, taking immense amounts of admin time, so there have to be technical methods to help. And you can only get so far with automated dup-checking of uploads, using custom filetypes with custom editors which track attribution history, and so forth. So the hurdle is to make it harder to steal a graphic by decoding it, than it is to steal it by other possible methods (screenshotting, etc). All you're doing there is scrambling, you're not encrypting, because you can't legitimately call something encryption if you're handing the key to every possible recipient ever. And the end result is that you end up with an arms race, where you permutate, tweak and improve your scrambling a little each release. You can't even make it the very best scrambling that you could, because then once they cracked that, you'd not have anything further to move to in the arms race. Edit 1: So why not use an open algorithm? Well, with DRM, your only defense against cracking becomes security through obscurity. Using standard crypto is a significant disadvantage here, as it is explicitly designed to be as non-obscure as possible. It will be the first thing people try; there are plentiful libraries for them to use to make code to attack your data; and there is plentiful documentation and people with skill in using them to answer any questions the hackers might have. Edit 2: The question description says: "Obviously this phrase has been hotly debated and surely is only acceptable if discovering the mechanics of a proprietary algorithm doesn't actually reduce it's security." I do not believe that this argument holds water, in a logical sense, either for DRM or for real encryption. If revealing an algorithm doesn't reduce its security, then there is no reason not to reveal it. There is also every reason TO reveal it. Revealing has advantages, while concealing it has none. A concealed algorithm has no trust. If you reveal, you can also patent it, you are then protected from theft far better than just hoping that someone doesn't reverse compile it. You can also bask in the attention and adulation of the crowds for creating an impenetrable algorithm that was better than existing open solutions. Concealment just suggests that you have something to hide: that you feel you need the security through obscurity. Only DRM needs that, and that's only because "DRM encryption" is an oxymoron. - The answer is missing one thing: In the case of DRM, what makes proprietary crypto better than standard crypto? – Damian Yerrick Jan 16 at 1:53 @tepples Good call, I should make that clearer.... [edits] Added! Thanks! – Dewi Morgan Jan 16 at 2:34 I think the answer is in some ways very obvious and in some ways less obvious. The advantage is that if it's proprietary, then that means you're almost certainly paying another party for it and therefore holding them responsible for reviewing, maintaining, and debugging the encryption library when problems arise. Contrast this with a library like OpenSSL, which turned out to have serious hidden flaws for prolonged periods of time due to the lack of sufficient manpower and funding for its maintenance and review. (Note: I'm merely mentioning the advantages, not claiming that they outweigh the disadvantages or that there are no better alternatives.) - I read the question as being about secret algorithms, not just about being a commercial implementation that doesn't come with source code. – CodesInChaos Jan 16 at 11:00 @CodesInChaos: My answer applies to both cases though, doesn't it? – Mehrdad Jan 16 at 11:03 I only see it as an advantage of a commercial implementation over an open source one. – CodesInChaos Jan 16 at 11:29 I have to agree. Further, it is an advantage of a commercial implementation over a non-commercial one. An open source implementation can be commercial. – otus Jan 16 at 11:50 Keeping the Algorithm secure gives an advantage. Like GOST cipher was kept secret until its public release in 1994 whereas it was being used by Russians in 1970 too. So what we learn is, Algorithm must be vetted for security by experts and then efforts should be made to keep it secret from adversary. But just trying to keep a week cipher secret only provides false security because one never knows how and when information has been compromised. - Obscurity can work if your application is, indeed, obscure. If you're Microsoft, that won't work. If millions of dollars depend on the security, that won't work either. But don't make up your own encryption algorithm - chances are it'll be easy to break. Instead, modify an existing encryption algorithm that's already good. For instance, use AES but swap the 2nd and 7th bytes in each 16-bit block. Or swap the 2nd and 7th byte in the KEY. Then, you've got the security of AES, but it's not off-the-shelf AES. - I suspect this was downvoted because, if you want to stick something random in to make stuff harder for people using standard libraries to try and hack it, this is a poor way to do it, for several reasons, such as: it opens the possibility that you make a change which weakens it in some way; your code maintainers will never be able to update the library if it is updated, so will be stuck with whatever problems it has in the version you edited; when porting to other platforms you will need to edit the libraries for that platform, too; etc. – Dewi Morgan Jan 16 at 18:46 Instead, the usual approach is to add a secret - called a "pepper" (en.wikipedia.org/wiki/Pepper_%28cryptography%29 ) to each hash. Say, "{*+=;banana" .$UserID. This functions as both a salt and a secret change to the algorithm. So you're adding just as much security as if you changed the encryption library, but at a fraction of the work. – Dewi Morgan Jan 16 at 18:52
@DewiMorgan: This thread is about encryption, not hash functions. Besides, a concatenated "pepper" is better implemented with a real cryptographic construction, whether that's a MAC construction for a hash function (i.e. HMAC), encrypting password digests before storage, or (in newer password hash function designs) using the "pepper" as a server-side secret parameter (if available, e.g. Argon2's $K$ parameter). – Reid Jan 17 at 6:00
The security through obscurity you'd get through modifying the algorithm is easier gained through adding a layer to pepper the key as you pass it in to the algorithm. But you are right that for encryption, even this is flawed thinking. Any change, including a wrapper, can weaken the algorithm or library. The existing algorithms and libraries have been "proven" secure: it'd be hard to do the same for a modification. Reverse compilation is trivial, so what do you gain from the StO that is worth the removal of proven security? – Dewi Morgan Jan 17 at 18:49

protected by Maarten BodewesMay 30 at 19:45

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).