what could you do to make your custom algorithm as safe as possible
Try to re-use existing algorithms and research to the extent possible.
For instance, suppose you are designing a block cipher (for some reason), and you need some source of non-linearity. Some researchers have already gone to the trouble of finding s-boxes with optimal cryptanalytic stats and minimal implementation costs. There are analogous results for linear layers as well. If you incorporate the results of these works into your design, you can rest assured that they will probably not be the weakest links in your design.
At a higher level, you could also simply re-purpose an existing primitive such as ChaCha and make a hash function out of it (like BLAKE). Or you could re-purpose an existing hash function into a (inefficient) stream cipher, like many others have done/tried to do already.
You (or someone with the same question) might say:
"But wait, if I do those things then I'm not really coming up with something of my own!"
I think it is safe to say: Symmetric cryptography is pretty much a solved problem by this point.
You can play around with it if you want, but don't expect to come up with anything (good) that hasn't been done already.
A note about "security"
You need to define your threat model before you can conclude what is/is not good enough in regards to security.
If your only goal is to keep your messages confidential from your grandparents, then you can probably get away with a pretty low quality design (unless your grandpa happens to be Adi Shamir or someone similar of course).
If your goal is to keep your messages safe from nation-state level actors with billions of dollars for their budget and hundreds (or thousands) of cryptographers and mathematicians at their disposal, as well as good old fashioned thugs, then you should probably think twice about the expectations of your design.
- If this is your threat model, this is where you need to really stick to pre-existing and battle-hardened constructions. An "oops" in your design could cost you or one of your friends/users dearly.
A secure design and a secure implementation are not the same thing. Coming up with a design that is not broken is only half the battle - you also need a way to implement the design in a way that can't be exploited, and this is a non-trivial subject. Just knowing how to program does not mean that you can create a secure implementation of an algorithm.
If you read all of that and are still interested in doing it, please remember one thing:
- Don't advertise your construction for others to use (outside of your friends who you have of course already informed that it is your own home-rolled construction and could break at a moments notice with no warning... right?).
If you set up a website preaching the virtues of your algorithm, the only people who would use your construction are people who don't know any better. This would be taking advantage of peoples ignorance at the expense of their safety for no good reason. If you did something like that in real life as opposed to cyber space, you would probably go to jail.
*Take my word for what it's worth, I'm just someone that likes cryptography.