I have been looking into the SHA-1 algorithm. I found that there are a set of functions and constants in the algorithm that have been standardized (section 5 of RFC 3174). If I want to use SHA-1, which I know is not advisable, does changing the constants and altering the functions have a positive effect on the security of the algorithm, assuming the attacker does not now of this alteration?
Changing values of the constants and / or the logic of functions from SHA-1 would just create a different algorithm. Depending on how large the changes are you could then say that the altered algorithm is similar or belongs to the family of "SHA-1-like" algorithms.
The effect on security:
We just don't know. You could of course try to do some large-scale calculations, i.e. try to see if known attacks of SHA-1 still apply to "altered-SHA-1", or try to find collisions, look for indications if the avalanche-effect still is good enough and so on.
It's not good practice to rely on security by obscurity, for multiple reasons:
- In your case the security of your model is based on the fact that the attacker would never gain information of your used algorithm.
- You can't be certain that your altered-SHA-1 is even more secure than the already vulnerable SHA-1. And because you haven't released this algorithm to the public you would have no idea if it's vulnerable.
In this case it's good practice to follow Kerckhoffs's principle.
I wouldn't suggest heuristically changing constants. However, there is a method with rigorous foundations that considered using randomized hashing in digital signatures. This method has the advantage that one can assume less from the hash function, and thus if the hash function is not "fully" collision resistant, the signature scheme can still be secure. In the method described in Strengthening Digital Signatures via Randomized Hashing by Halevi and Krawczyk, one only needs target collision resistance, which is much weaker than full collision resistance. Of course, this only helps in the context of signatures.
I did some more research about the topic and someone did some experimenting on this. He found that when changing the constants for the initial state, the security of SHA-1 is compromised.
It is an interesting read and is indeed a good answer to my question. One should not alter the constants because it can affect security in a negative manner. He showed that he could choose constants such that particular files collide. If I were to randomly alter my SHA-1 function, I might create these collisions that can be exploited, though unlikely it still can happen.
This is the webpage of the Malicious SHA-1 project, a research project that demonstrates how the security of the SHA-1 hashing standard can be fully compromised if one slightly tweaks some of the predefined constants in the SHA-1 algorithm. That is, we show that systems using “custom” versions of SHA-1 may include backdoors exploitable by the designers. Such custom versions of cryptographic standards are typically found in proprietary systems as a way to personalize the cryptography for a given customer, while retaining the security guarantees of the original algorithm.
How are the modified constants weaker? They aren’t.
Unlike operational parameters such as rotation values, the exact values of the round constants we modify do not have a direct impact on the security of the hash function (although there are some values that would weaken the algorithm by introducing undesirable statistical properties, for example choosing all 0s).
Our attack only exploits the fact that the designer has freedom in choosing the constants, not any particular weaknesses. To third party analysis, malicious SHA-1 remains as strong as the original SHA-1: the backdoor is “undiscoverable“, it can only be exploited by the designer.