I have been academic papers about Rijndael, Serpent, and Twofish, and there is this term that is vague to me. I cannot find a tangible definition in google. Can someone briefly define, describe, and/or give examples about security margin and where it is used.
In this context 'security margin' is a measure for how much better we need to get at analyzing a cipher to break it. Such advances in cryptoanalysis require new ideas of how one might attack a cipher. Thus estimating how strong a cipher is, is hard.
Ultimately we can only tell something is broken, after we've broken it.
We typically look at a few properties to estimate the security margin:
So if we put a lot of effort into analyzing some cipher, think we understand it well, and still only broke a few rounds, then we believe the security margin is large.
Resistance to brute-force can be easily quantified, estimating resistance to cryptoanalysis is full of guesswork and pretty subjective. So it's not surprising that you only find pretty vague statements.