I am new to crypto subject. I am studying the confusion and diffusion in AES etc. Is there any general quantitative method to measure these properties. Let us say Algorithm A produces cipher having more confusion/diffusion with respect to Algorithm B. How can one quantify these criteria? Any specific Model or formula for same?
Diffusion can be quantified by examining the branch number of the diffusion layer. The branch number indicates how much a difference in the input will spread around the rest of the state after the transformation has been applied.
The branch number can be used to prove lower bounds on the amount of diffusion that the permutation provides.
Measuring avalanche statistically can only tell you if your diffusion is awful, it can't really be used to prove that diffusion is complete and even throughout the state, or that one design is better then the other (assuming that both have decent diffusion).
Confusion, or non-linearity, can be quantified by measuring stats such as differential and linear probability. For small enough mappings, it is easy to take these metrics by simply building a difference distribution table and linear approximation table and simply examining the results. For larger mappings this may prove challenging if not intractable.