I'm an undergraduate math student, and I'm thinking about topics for an undergraduate thesis. I have an interest in topological spaces and number theory, but I'm having trouble identifying a relationship between those and cryptographic theory. In On Post-Modern Cryptography, Goldreich talks a lot about rigorous analysis. But it's still not giving me a clear understanding of what type of math a person doing research in theory does in general. Is cryptographic research in the same line as formal mathematical proof? Is there a need for abstract mathematics in modern cryptography? I can program, but application is not really my interest and it would take too long in my degree to add a formal computer science aspect.
Is cryptographic research in the same line as formal mathematical proof?
Not always, but the large area of provable security is. Provable security is (essentially) the task of formally proving theorems about crypto, which will in general be of the form "if this mathematical hypothesis is true, then this cryptographic primitive is secure against a given type of attack". Each time a new primitive is suggested to address some security requirement, it comes with proofs of theorems providing hindsights regarding the security of the scheme (though that's not always the case, especially with papers from the early days of crypto). An important part of cryptography is also about analyzing and implementing cryptographic protocols and primitives, but that does not mean all cryptographers do also implement crypto. In many labs, 95% of the work will involve essentially pen-and-paper analysis - although even the most theoretical works sometimes lead to result that one might want to implement, if only to see how efficient it is.
Is there a need for abstract mathematics in modern cryptography?
Yes and no. A knowledge of the formalism in some area of abstract mathematics can help a lot into understanding and studying crypto, but in most cases, you will not need a deep knowledge of the most advanced results in abstract mathematics. For example, most results you would learn from an advanced math course about elliptic curves would be outside the scope of the background needed to understand elliptic-curve-based cryptography. Similarly, you would not need extremely advanced courses on algebra to understand the notions used in crypto (essentially, standard algebra results on finite fields and additive groups suffice for many things, plus some more crypto-related algebra result).
However, as is common in theoretical computer science, mathematics often show up unexpectedly when studying some problems, and relatively deep mathematical results can prove useful in the analysis of some cryptographic primitives. The first example that comes out of my head is this recent paper from ePrint (the crypto archive) which establishes lower bounds for a primitive known as order-revealing encryption (an encryption scheme that does not hide the result of the comparison between encrypted messages), and relies on (an extension of) a theorem of Erdös on tournament graphs.
The high level bit is: studying math is a very pertinent choice if you're planning to do theoretical crypto later, because being at ease with mathematical formalisms will make you way more comfortable than many with crypto. It is widely acknowledged in crypto that the field would benefit from having more mathematicians studying this area. However, do not expect your knowledge of advanced math results to be crucial to your work in crypto on a daily basis, and theoretical crypto in itself remains a science with its own formalisms and practices that you would have to study if you want to work in the field.