Rolling your own crypto

Question

I know this is usually frowned upon, but supposing you were to roll your own encryption cipher (completely on your own or with a small group of friends, with no peer review possible), what could you do to make your custom algorithm as safe as possible (or rather, to keep the danger of using it to a minimum)? Similarly, what would you do for something like a hash function to make the probability of it having some trivial collision or preimage attack as low as possible?

I understand that there would be no way to approach the security of algorithms like AES, and I have no plans to use custom algorithms for anything important, but I thought it would be informative to think of what some best practices would be for rolling your own cryptosystem if you were to do so.

Answer 1

what could you do to make your custom algorithm as safe as possible

Try to re-use existing algorithms and research to the extent possible.

For instance, suppose you are designing a block cipher (for some reason), and you need some source of non-linearity. Some researchers have already gone to the trouble of finding s-boxes with optimal cryptanalytic stats and minimal implementation costs. There are analogous results for linear layers as well. If you incorporate the results of these works into your design, you can rest assured that they will probably not be the weakest links in your design.

At a higher level, you could also simply re-purpose an existing primitive such as ChaCha and make a hash function out of it (like BLAKE). Or you could re-purpose an existing hash function into a (inefficient) stream cipher, like many others have done/tried to do already.

You (or someone with the same question) might say:

"But wait, if I do those things then I'm not really coming up with something of my own!"

I think it is safe to say: Symmetric cryptography is pretty much a solved problem by this point.*

You can play around with it if you want, but don't expect to come up with anything (good) that hasn't been done already.

A note about "security"

You need to define your threat model before you can conclude what is/is not good enough in regards to security.

If your only goal is to keep your messages confidential from your grandparents, then you can probably get away with a pretty low quality design (unless your grandpa happens to be Adi Shamir or someone similar of course).

If your goal is to keep your messages safe from nation-state level actors with billions of dollars for their budget and hundreds (or thousands) of cryptographers and mathematicians at their disposal, as well as good old fashioned thugs, then you should probably think twice about the expectations of your design.

• If this is your threat model, this is where you need to really stick to pre-existing and battle-hardened constructions. An "oops" in your design could cost you or one of your friends/users dearly.

Note

A secure design and a secure implementation are not the same thing. Coming up with a design that is not broken is only half the battle - you also need a way to implement the design in a way that can't be exploited, and this is a non-trivial subject. Just knowing how to program does not mean that you can create a secure implementation of an algorithm.

Finally

If you read all of that and are still interested in doing it, please remember one thing:

• Don't advertise your construction for others to use (outside of your friends who you have of course already informed that it is your own home-rolled construction and could break at a moments notice with no warning... right?).

If you set up a website preaching the virtues of your algorithm, the only people who would use your construction are people who don't know any better. This would be taking advantage of peoples ignorance at the expense of their safety for no good reason. If you did something like that in real life as opposed to cyber space, you would probably go to jail.

• Advertising your design will probably just put you on a list of snake oil salesman and destroy your own reputation.

*Take my word for what it's worth, I'm just someone that likes cryptography.

Answer 2

No answer could possibly start without referring to Why is writing your own encryption discouraged?. But you seem to be aware of those restrictions. There are several senior contributors here that roll their own, but correspondingly they have experience in appropriateness and threat model evaluation. Nevertheless, I can think of three examples where very strong security can be achieved.

1. Your own wide block cipher. There is the concept of a generic Feistel network what can be of an arbitrary number of plaits. If you used standard AES, it would be possible to go wide with it, following accepted cryptographic principles. You would need to determine a number of acceptable rounds. 3 - 4 works for a typical two plait Feistel network. I've not managed to determine the required number from papers I've seen, but you could research this more thoroughly and take a conservative approach. Just make it a couple of hundred if you can afford the time. You would have to also develop some form of key schedule, but using a secure primitive makes this much simpler. 8 AES' would create a 1024 bit wide cipher that would ostensibly be secure. All of the analysis already exists.

2. Key derivation function that is hard in time and space. Again building upon a secure hash function, it's simply a programming challenge to find a non paralliseable construct. 10 weeks and /or 10 GB should be reasonably possible. Just use enough arrays , lists and rounds.

3. Hash based randomness extractor. Not a strictly crypto construct, yet on topic for this forum. Since there is no merit in reversing an extractor function for a TRNG, all you need to develop is the avalanche effect. And that is helped anyway by the pigeon hole principle. Such a construct can be automatically tested with randomness testing tools and does not require crypanalysis.

If you're looking to parallel something like SHA or AES directly, you're in for a long haul. It would be safe to say supplanting them is impossible for virtually all terrestrials due to the reasons in the first link.