There is a fundamental difference between designing your own algorithm (e.g., block cipher) and designing your own mode of operation. In particular, it is possible to formally prove the security of a mode of operation, and this is not possible with a block cipher. Thus, if you know how to prove security, then there isn't any reason not to do this. (Of course, you should in general have a reason for doing something that isn't currently covered by any standard. But, there are reasons that come up.)
Personally, I would never try to design a block cipher or the like, since this requires many years of expertise and even then is really hard. Even block ciphers designed by experts have flaws, so it's best to just use AES etc. However, I would design a mode of operation if I had a reason to. But then again, I have years of experience writing proofs. If you don't, then I would recommend to stay clear of this as well.