What factor is preventing cryptography based on chaos theory from being of practical use? Is it because of the unpredictability of the system?
Kodlu's answer succinctly addresses the practical challenge with digital systems, but a form of chaotic cryptography is used in analog. Chaotic oscillators are used to encrypt analog video streams, ie: http://www.dtic.mil/get-tr-doc/pdf?AD=ADA503411 Unfortunately, most of the papers are paywalled; however, the fundamental mathematics are based on reductions of Chua's Circuit.
I've seen the results and the pictures are grainy, but the brain can figure it out. Also, I know that these techniques have been used to retrofit old analog modems, but I've never seen one in practice.
All chaotic crypto papers I have seen lack rigorous security analysis, their main shortcoming being that there is no understood mathematical machinery of the type used in non-chaotic crypto such as:
- low differential probability
- low linear bias
- guaranteed period, window, nonlinearity properties
Also, most chaotic systems are continuous, need infinite precision real numbers to represent, so since sampling/quantization will be needed, some of the properties of chaos may be lost.
Most chaotic crypto papers appear in the general physics/engineering literature, and are not even seen by most crypto researchers.
So, why would anyone use them when there are well understood techniques with long history of resisting cryptanalysis. The chaotic crypto authors display computational evidence (equidistribution only, in most cases) and quit.
If they dont properly cryptanalyze their designs no one else will. However, typical chaotic systems are almost impossible to analyze theoretically, their state trajectories are uncontrollable, by definition.