I want to add some custom header to encrypted file to identify my encryption,consider safe and reliable.like pseudo format extension or file size.Is there any algorithm or tool to make it?
As indicated, the header format of your proprietary encryption is entirely up to you. Still, as I see this forgotten or done imperfectly a lot of times, I'll give some general hints in the right direction.
First of all, complexity is the enemy of security. If you make your header overly complex you may get into trouble. It could be possible to generate invalid packets, buffer overruns because of incorrect length fields or simply combination that do not make sense.
Furthermore you must be aware that attackers may be able to mess with your header format. This is mainly a problem for encryption formats that are or can be used for transport security. For transport security you should take care of man-in-the-middle attacks. So if an attacker can downgrade your security level through the header format then that's obviously a bad thing. You don't want to end up with all the configuration options for, e.g. IPSec. It makes no sense to allow (and code, and test!) AES-256 with MD5 configured together.
If you have lots of configuration options then those are likely to have a big influence on security if they haven't been established beforehand. So you must make sure you check your header format in advance. It's highly recommended to let your header format be included in any authentication tag calculation (MAC), and to make sure that that MAC verification itself isn't an issue. The simplest attack is simply to remove the MAC authentication tag entirely if your protocol supports it. This sort of attack actually happened on protocols like WS-Security (web services) due to broken implementations of both Glasfish and Apache/Tomcat where the signature was simply stripped off and the server accepted those messages without issue.
Probably the simplest header is a single identifier defining the version of the protocol. That version then defines a well defined suite of algorithms with parameters that make sense. So for instance byte version
01 would indicate that AES-GCM is used with a security level of 128 bits. The security level of 128 bits is then maintained for all other algorithms as well (e.g. key derivation with 256 bit hashes and RSA with 4K bit keys.
It may be useful to define an additional 8 byte constant value so you can identify the file type by looking at the content instead of the file extension (if applicable, e.g. for in place file encryption).
So there you are:
- apply the KISS principle, don't allow overly flexible configuration;
- make sure you are protected against attacks on the header and that buffer overruns are not possible (or at least warned against);
- verify the validity of the header upon receival, checking as much as possible, and at least an authentication tag;
- include a constant value for file identification when applicable.