How to check the strength of an encryption algorithm?

Question

I asking for advice on how to check strength of an encryption algorithm.

Answer 1

There are still a lot of unknowns surrounding your question, so I will do my best to answer and fill in the holes as necessary.

One big question that remains is whether you are using already well established cryptographic building blocks (AES, SHA-2, RSA, etc) or whether you have invented some new building block (as CodesInChaos put it "AES is for wusses, use Samuel's super secure cipher"). I'll attempt both.

For the first, the most common way to go about this is through modeling. You model your protocol or setup in some way as to simplify the analysis so that certain properties can be proved. For an example of how this is done, see this paper which proves security for SSH. I'll give you this warning though, modeling necessarily simplifies things and can abstract away flaws. In which case the flaws won't be discovered in the process. This is evidenced by this paper which shows a flaw in SSH that had been abstracted away in the previous analysis. There are also tools to help automate this process as described in this other question.

Instead, assuming the second option above, the best way to begin analyzing Samuel's super secure cipher is to write up an academic-level research paper and publish it in a top journal or conference. See IACR for some pointers. Once all the details are published, the research community will hopefully get involved. The reason for this is Schneier's Law: "Anyone, from the most clueless amateur to the best cryptographer, can create an algorithm that he himself can't break."

Note
Many here would say "never design your own". This is very sound advice for real world applications. The only reason I can ever see someone designing their own cryptographic procotol/cipher/scheme/etc is for educational purposes. Only after you have done this for a decade or more, with close mentoring from an established cryptographer, should you even think about designing for a production system. And even then, the advice about publishing still holds.

So, assuming this is for educational purposes, go ahead and design your own, try to publish it if it is something novel or at the very least run some of those analysis/modeling tools on it. But never, never fall to the urge to use the thing.

Answer 2

As many have said, the short answer is probably "no". Not because you asked the question in a wrong way but because the fact that you ask this question implies that you don't know what sort of attacks your application would have to be resilient against. Thus, there is very little hope that you managed to achieve your goal.

Possible attacks your application would have to be able to withstand are linear attacks, differential attacks (and their parents)... And I'm only talking about attacks on the idealised algorithm, not its actual implementation! Then you need to watch out for attackers trying to access memory cells they are not supposed to and other side-channel attacks.

The bottom line is: it is extremely complicated. Even if you use a known algorithm (like the AES), it is very easy to mess up the implementation. That's why you should always use standard libraries which encapsulate all the solutions to these problems or, even better, complete programs (opengpg, openssl...) written by teams of (very) knowledgeable people.

Answer 3

The first step is to write a new question on crypto.stackexchange.com asking for advice on how to check whether if it is secured enough (or not).

The second step is to stop, throw away everything you've done and become an expert in GPG and then port GPG to your platform.

The rationale for being so discouraging is that if you have to ask how to check whether it's secured enough you obviously don't know nearly enough to even start writing an app that encrypts text securely. Many people who would have a very good idea about how to check the security would balk at the task of writing something like GPG themselves.