The question itself is not precisely well formulated: cryptographic algorithm design is, like most scientific process, is a process of searching for theoretical solution (algorithm) to theoretical problem (set of desired security guarantees), which then has to be proven to be proper solution for the problem (security proof), and only then get implemented in code. Methods of doing such research have been attempted to be formalized many times, but I'm not aware of any good results. Like most hard science, it's thinking, writing formulas, trying and retrying, incrementally improving not some external objects, but author's understanding of the subject, which finally, after Nth attempt translates into cryptographically sound algorithm. Or what seems to author to be one, only yet to be proven faulty by others (see Schneier's law).
However, if you step away from details far enough, it will have some similarity with traditional Waterfall process, because:
- Inception. There are strong formal demands towards algorithm implementations and algorithms themselves; coming up with a solution / algorithm / superposition of algorithms (cryptosystem) can be considered 'writing a technical specification'. However, while in software specification and reality can mismatch a bit, and specification is considered to be a 'general guiding document', in cryptography specification is actually 90% of work in terms of importance. Writing proper crypto code is much less hard when you've got right algorithm with right demands and proper proofs, and is simply pointless when you don't.
- Implementation. When the algorithm is properly described, it gets implemented in a typical waterfall fashion: there isn't space for breaking down things into iterative improvements (which is prerequisite for sucessful Agile process).
- Integration. However, writing integration layer around the core crypto code can be iterative process, and while formal acceptance criteria are much more harsch than in software development (you can't "almost" make it, you either pass the tests / proofs or you don't), the process itself is easily dissectable into sprints, etc.
Consider this a private opinion of person, who participated a lot in applied cryptographic research, design and development, yet never suggested a fundamental algorithm himself (because see Schneier's law).