# After implementing a novel encryption algorithm, how would one go about analyzing its security or get help from others in doing so?

Preface: This question was originally asked on Theoretical Computer Science and later on Computer Science. The kind people on cstheory.SE referred me to this web site. It is being repeated here in an attempt to find a satisfying answer.

Over the years, two novel encryption techniques have come to mind and been implemented as programming libraries that could be integrated into applications. However, how to analyze their security and vulnerability characteristics has never been very clear, and their usage has been limited to mainly experimental tests. Are there tools available for automated examination of such parameters one may be interested in understanding for an encryption library? Are there bodies of people who are interested in being introduced to new encryption concepts for the purpose of executing their own personal analysis on such a process? I'm not sure where to look.

The first encryption algorithm is a mono-alphabetic simple substitution cipher. It requires two keys to operate and is designed to frustrate frequency analysis. The longer of the keys forms a table by which plain-text has a normal substitution cipher applied. Each encoded byte is then split into four values of two bits each. The second, shorter key is then used to allow a random selection from four groups of sixty-four unique bytes each. Each two bit value from the encoded byte is used to select which group of sixty-four bytes to use. Encoding has two disadvantages: the output is four times larger, and repeated data encoding may allow some frequency analysis.

The second encryption algorithm is a stream cipher like the first but internally operates on blocks of data. It utilizes two keys to operate: the first is a two-dimensional array that describes how to construct a (virtual) multidimensional grid, and the second is an initialization vector for the encoding/decoding engine. It attempts to overcome frequency analysis by encoding bytes with a window of preceding bytes (initialized from the second key). A byte with its preceding window of bytes form a multidimensional index into the aforementioned grid. Unfortunately, encoding duplicate blocks of data longer than the window size starts yielding equivalent data.

-
If you're considering them as an actual security measure, forget it. Crypto is a field where "rolling your own" is a very bad idea. – Polynomial May 24 '12 at 20:54
What is the simplest method of getting years of peer review and revision for a new cryptographic algorithm? I am trying to find out where to start. – Noctis Skytower May 24 '12 at 21:34
If you want to learn how to be an expert in cryptography, the way to learn is not to design cryptoalgorithms; instead, the way to learn is to break cryptoalgorithms. If you want to get professional review of an algorithm, well, the most reliable way would be to pay for it; however, it's unlikely you'd learn much from it (at least, you won't learn nearly as much as you would if you broke it yourself). – poncho May 24 '12 at 21:34
If frequency analysis is your top threat, you're at least 50 years behind the times. – CodesInChaos May 24 '12 at 22:24
Shows you how little I know but also why I want to learn more. :-) – Noctis Skytower May 25 '12 at 13:31

To answer your question "After implementing a novel encryption algorithm, how would one go about analyzing its security", well, the easy answer is "no, it's not secure". The reason I can confidently give that answer is that you're a newbee, and newbee ciphers are never secure (unless they are massively overcomplicated; it doesn't sound like that's the case). A newbee is about as likely to create a secure cipher as a blind man would be to paint a good piece of art; neither has any reference as to what works and what doesn't.

However, I assume you are looking for a more detailed answer than that. Well, that gets tough for a newbee. There are no automated tools that can validate an algorithm as "secure" (as secure means that no one can come up with a creative method to attack it, and automated tooling has difficulties with "creative").

So, you would need to have a human look at the cipher. You could try to get an expert cryptographer to look at it, however (and you might find this shocking) there are no groups of expert cryptographers that are eager to attack ciphers from newbees. And, even if you do get someone to look at it, you'll run into the next difficulty - they'll point out exactly how to break it; you'll tweak the cipher to stop that exact attack, and you'll have no idea whether tweaked cipher is any stronger than the original (except for that exact attack); indeed, the tweaked cipher could easily end up being weaker.

The alternative is to invest the time to become an expert yourself; the way to do that is to break ciphers yourself. Yes, that's quite a commitment (becoming an expert in any subject is pretty much by definition hard work); but if you are serious about designing ciphers, that's what you have to do. You might start with Stanford's course (note: I haven't gone through it myself); Schneier has an intro-to-cryptanalysis course here; or you might want to start by attacking your own ciphers.

The last isn't as silly as it sounds; here's a few suggestions:

• Can you attack a 'reduced strength' version (say, with a reduced 'byte size')?

• Suppose that the attacker can encrypt messages of his choosing (which is actually a standard cryptographic assumption)? How can he use that? For example, for your first method, suppose he asks for the encryption of thousands of repetitions of the same character? What can he deduce from the ciphertext?

• Once you can figure out how to break it with chosen plaintext, how can he attack it if he knows some of the plaintext (but obviously not all)? How can he attack it if he doesn't know any of the plaintext, but he does know that it is, say, English-ASCII?

To give you a target, I suspect that against your first method, I could decrypt the encryption of perhaps a 10k English-ASCII message, or a 200 byte English-UCS-2 message. And, no, I won't tell you how; that's for you to discover (hint: why do I need so much less text with UCS-2?)

-
I suppose you mean UTF-16 (or UCS-2) instead of "Unicode" in the last paragraph. "Unicode" is not an encoding to bytes, just a set of characters. (And most English text looks same in UTF-8 like in ASCII.) – Paŭlo Ebermann May 25 '12 at 20:06
@PaŭloEbermann: yes, sorry about that - my only exposure to Unicode is, in fact, UCS-2, and so I tend to forget that's not the only encoding. I went ahead and corrected the response. – poncho May 25 '12 at 20:10
Would you be interested in trying to decrypt one of the messages that you proposed? I could provide a ten-thousand byte message and two-hundred byte message for you. – Noctis Skytower Jul 2 '14 at 13:08
@NoctisSkytower: not off hand. You may find this shocking, but I'm not eager to attack a cipher from a newbee. Have you tried analyzing the cipher yourself? Have you looked into (for example) potential chosen ciphertext attacks (which is an easy place to start)? – poncho Jul 2 '14 at 13:40
I have not been twiddling my thumbs for the past two years, if that is what you mean. – Noctis Skytower Jul 2 '14 at 13:41

First, why did you invent your novel algorithms? What problem were you trying solve, what are the advantages and the trade-offs of your algorithms? I assume you have read Bruce Schneier's Applied Cryptography and can compare your algorithms with others.

You can try to get your algorithms published in a crypto journal or presented on a conference, like those published and organized by IACR. If you're a beginner, that's probably too high for you. Try smaller journals and local conferences. And even if you get rejected, you'll likely get useful feedback concerning the algorithms---first peer reviews.

As commentators above noted, it's not easy to enter the crypto community nowdays, but it's possible. Some useful ideas were actually quite simple and straightforward, for example Shamir's Secret Sharing, Visual Cryptography, or CAPTCHAs.

-
Why? I have been programming for fun over the past ten years while studying the subject in school and college. As a child can learn much from playing, constantly trying to learn more and exercising that knowledge has expanded my understanding in computer science. A brief, shallow study of cryptography encouraged me to play with the concept; but like any student, instruction from a more knowledgeable person is needed. When I know what is wrong with my designs, wisdom can be applied to any further activity in the field. – Noctis Skytower May 25 '12 at 13:55
I don't think your motivation would help you get your algorithms published. What conferences and journals like to know is why someone else should be using your algorithms. So I suggest you look hard into your and other algorithms and try to find any competitive advantages your algorithms might have. – Igor F. May 25 '12 at 16:52
Would the facts that (1) it can selectively encrypt certain bytes and ignore others and (2) it can encrypt a block of bytes of any size be competitive advantages? To clarify: (1) Someone may only want to encrypt bytes having the values 32 - 95. Maybe the person only wants to encrypt printable bytes. It might be advantageous to encrypt all but null bytes. (2) The encryption strength is easily configurable. If you want to encrypt a block of 10 bytes at a time, you can. If someone else wants the encryption window to be 10000 bytes, it is possible. The block is considered for each encoded byte. – Noctis Skytower Jul 1 '14 at 21:48