2
$\begingroup$

Today I implemented SHA-1 in JavaScript. While doing so I stumbled upon two requirements which are not mentioned in the RFC-3174.

  1. Text is supposed to be utf-8 encoded / one character should be 8 bit long. This makes sense, since the same text should always return the same hash, but I couldn't find anything about it in the RFC. Is there any place where this is documented?

  2. For some reason the bytes for each 32-bit word need to be reversed so the first byte comes last. e.g if we have chunk of 32-bit (reading from left to right).

    00110101 00110011 01100110 00001111
    

the bytes should be arranged like this, before processing them:

    00001111 01100110 00110011 00110101

I also couldn't find anything about this in the RFC. Where was this defined and is there a reason why this needs to be done?

$\endgroup$
5
  • 3
    $\begingroup$ What source is telling you that SHA-1 requires UTF-8 text? Because it doesn't—SHA-1 treats data as bytes, not text. $\endgroup$ Aug 29, 2016 at 22:34
  • $\begingroup$ The official definition of SHA-1 is there $\endgroup$
    – fgrieu
    Aug 29, 2016 at 22:43
  • $\begingroup$ #2 is quite well known, and is also required with SHA256 and SHA512 and their derivatives. If you want to avoid that, go with Blake2 $\endgroup$ Aug 29, 2016 at 23:55
  • $\begingroup$ @LuisCasillas Well when comparing the results of my sha1 implementation and the results for the same text content from various other sha1 tools and some examples from Wikipedia, showed that my results are different from everybody else. So it seems that it is "normal" to use utf-8 when hashing text. $\endgroup$
    – Jovan
    Aug 30, 2016 at 4:53
  • $\begingroup$ @TitanNano, I would suppose most examples are actually just ascii, i.e. plain English 8 bits/character. If you see UTF-8 used, it's just because it's quite widely regarded as a useful encoding precisely because it's compatible with ascii. (UTF-8 characters can be up to four bytes) $\endgroup$
    – ilkkachu
    Aug 30, 2016 at 8:07

1 Answer 1

6
$\begingroup$

Text is supposed to be utf-8 encoded / one character should be 8 bit long. this makes sense, since the same text should always return the same hash, but I couldn't find anything about it in the RFC. Is there any place where this is documented?

This isn't a requirement of SHA-1, but rather appears to be some requirement of the context in which you encountered this implementation.

SHA-1 doesn't assume anything about the bytes in the message; they may be text in any encoding, image data, random bytes, whatever. However, an actual user of SHA-1 who is hashing text would care about this. Suppose two applications have the same text string, but one uses UTF-8 internally and the other uses UTF-16. If each one just hashes its own raw memory bytes they're going to get different results even though their strings, textually, are the same.

For some reason the bytes for each 32-bit word need to be reversed so the first byte comes last. e.g if we have chunk of 32-bit (reading from left to right).

This is an endianness conversion, so you should familiarize yourself with that concept. This is in fact a requirement of SHA-1. FIPS 180-4 says (p. 7):

Throughout this specification, the “big-endian” convention is used when expressing both 32- and 64-bit words, so that within each word, the most significant bit is stored in the left-most bit position.

The RFC that you link says it this way (which is more confusing, I think):

The least significant four bits of the integer are represented by the right-most hex digit of the word representation. Example: the integer 291 = 2^8+2^5+2^1+2^0 = 256+32+2+1 is represented by the hex word, 00000123.

Whether you need to perform the step that you describe is context-dependent. For example some CPUs are natively big-endian and wouldn't need that step—the bytes would already be in the correct order for the 32-bit operations that SHA-1 performs. Others are natively little-endian and would need the conversion. It can also depend on the details of some network protocol or library that you're working with.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.