I have some code that accepts a password in string format, and hash it with SHA3-512 before passing it to bcrypt for hashing. However, by some coincidence, I found a test case that produces a hash that contains '\x00' in the middle of the hash result, as seen below:

password_str = 'tes15!tes15!tes15!tes15!tes15!tes15!tes15!tes15!tes15!tes15!tes15!tes15!.'
password_bytes = password_str.encode('utf-8')
hashed = hashlib.sha3_512(password_bytes).digest()

The hashing result that contains '\x00' in the middle of the hash:


As we can see, there is an '\x00' found in the middle of the hash, but not at the end. bcrypt detects this a ValueError exception:

if b"\x00" in password:
    raise ValueError("password may not contain NUL bytes")

I understand that '\x00' is a reserved character and possible attack vector due to hash-length extension attacks. However, I do not think that I have done anything outside the ordinary. Is there a best practice that sanitizes the hash prior to passing it to bcrypt that I am missing, or is there another type of byte-encoding that should be used with passwords?

It would be weird to reject a normal password just because it's derived hash conflicts with the auth library used.. Any help or advice is greatly appreciated. Thank you!

EDIT: After posting, I found some issues on bcrypt that describes my problem https://github.com/pyca/bcrypt/issues/55 https://github.com/pyca/bcrypt/pull/57

  • $\begingroup$ I don't know why bcrypt would stop it, whether the limitation exists in the spec or it's a bug/feature of the library. Why are you passing a hashed value into bcrypt in the first place, though? $\endgroup$ – Aman Grewal May 24 at 16:59
  • $\begingroup$ This is a good question for Stack Overflow with minimal complete and a runnable example. Do you want it to migrate? $\endgroup$ – kelalaka May 24 at 17:00
  • $\begingroup$ I was following some practices used in the dropbox example here: dropbox.tech/security/… . It is also to ensure that the incoming bytearray length is <72 (64 in the case of hashing using sha512) as bcrypt only allows a max of 72 incoming bytes! I'm a novice so I might be wrong.. $\endgroup$ – Benji Tan May 24 at 17:01
  • 1
    $\begingroup$ @kelalaka Sure :) sorry if this is the wrong forum to post, as I was looking for errors in my process that resulted in an '\x00' bytesequence appearing in my hash. It is possible to mitigate by using other encoding types (e.g. byte64, ascii, etc) but I wanted to know the best practice used to encode passwords before hashing too! $\endgroup$ – Benji Tan May 24 at 17:04
  • 1
    $\begingroup$ I see some discussion about on-topicness, but "should a bcrypt algorithm implementation allow NUL-bytes" seems pretty much on topic to me. If required we can somewhat rephrase the question, but as for now I'm all for having it stay. $\endgroup$ – Maarten Bodewes May 24 at 21:26

You should not be hashing it before passing to bcrypt, which is designed to do the hashing and key-stretching work itself.

It's choking on the hash result because it's expecting a redundant, mushy, ASCII (or UTF-8), not-rigorous, user-entered string.

Generally speaking, hashing things that might be untrustworthy is good to avoid various numeric vulnerabilities (e.g. the internal SHA-512 ops in Curve25519 to sanitize things) -- but, in this case, you can just trust bcrypt; it's designed to work safely and well when fed a lame, user-created password (and a proper, randomly-generated salt 😉).

🤔 On further contemplation, though, I believe that the behavior of pyca/bcrypt is actually incorrect here. You are supposed to use a cryptographic hash function to convert passwords which are more than 72 bytes (576 bits) into a size that bcrypt can accept. [This is only for pigeonholing purposes; it IS still designed to safely stretch cryptographically mediocre passwords.]

Looking at the ticket on their tracker, and the changeset which they affirmed as having "fixed" it, it appears that the authors did not actually fix their implementation's oddball behavior, but instead merely updated their documentation to recommend the following workaround:

a common approach is to hash a password with a cryptographic hash (such as sha256) and then base64 encode it to prevent NULL byte problems…

This seems sloppy and bizarre to me; it will, for instance, limit such digests' output to, at most, around 408 bits (with base85 encoding), thus truncating >72-byte passwords to just 51 bytes. I would be very interested to see the opinions of more experienced cryptographers on this.

This doesn't negate the first half of this answer, but it's something to keep in mind. Obviously, even with the workaround, the scheme is still quite secure (the best cryptographic schemes are designed to survive some slip-ups in their implementations); it just doesn't seem completely correct to be applying such arbitrary truncations to the user's password, above and beyond what the author of bcrypt itself applied.

You might want to consider bouncing that 72-byte limit back to your users, if you would otherwise have to truncate longer ones by a minimum of 30% to comply with pyca's implementation quirks. (I'm sure that those who care to max out their password's entropy and are hitting that 72-byte limit would, if they looked into how the sausage is made, ultimately prefer the former.)

  • $\begingroup$ Thanks for the answer! It makes sense to trust the library as it's also industry standard. I was just trying to implement a similar password encryption process used here: dropbox.tech/security/…. Also, the reasoning behind hashing prior to passing to bcrypt is to handle long passwords that are >72bytes in length (according to several sources). Are there any other ways to handle long passwords, as some password-managers do generate passwords that may exceed the limit of bcrypt. Thanks! $\endgroup$ – Benji Tan May 24 at 17:13
  • $\begingroup$ I have selected your answer as it answers my questions regarding pre-hashing and encoding. Cheers! $\endgroup$ – Benji Tan May 24 at 17:21
  • $\begingroup$ I thought about it as well but as I mentioned too, it would be odd if I reject a password because it does not pass the library's encryption methods, even though it is perfectly valid. I also attached their PR / Issues below as it pertains to my question too. One of the workarounds that I have read would be to truncate the resultant hash as you have mentioned, as hash values would be different for every input, however similar. Then again, sha256 and 512 are not that far apart for my purposes (school project, haha..). Security in architecture is interesting! Cheers! $\endgroup$ – Benji Tan May 24 at 19:36
  • 1
    $\begingroup$ @BenjiTan I have also updated my answer in light of those issues: for >72-byte passwords, you can either reject them, or hash-and-ASCIIify them. I would recommend the former, but they're both still pretty secure. (If pyca ever properly fixes the issue, then you should accept any length password and just hash >72-bytes ones.) In any case, there is no need to hash ≤72-byte passwords; they are still clear as-is. $\endgroup$ – JamesTheAwesomeDude May 24 at 19:38
  • 4
    $\begingroup$ Personally I don't think that it is up to the bcrypt implementation to test passwords. There are a lot of things that you could check w.r.t. to passwords, but the password hash is generally not the place to put a password policy into effect. Just disallowing NULL seems very arbitrary. It also disallows using other hashes, or extending the work factor by performing another bcrypt over a previous output. It should just accept bytes or random strings (+ encoding) if you ask me. $\endgroup$ – Maarten Bodewes May 24 at 21:22

Is there a best practice that sanitizes the hash prior to passing it to bcrypt that I am missing, or is there another type of byte-encoding that should be used with passwords?

One reasonable thing is to convert the hash to Base64 (say, with the standard base64 alphabet), and truncate to 64 characters (well below the most common input size limit for bcrypt, and not above the common threshold for LF insertion in Base64). 384 bits of entropy are aplenty. That fixes the question's issue, and more including malformed UTF-8, which some bcrypt implementations could check. It should insure portability across implementations that process at least the first 64 characters in the standard way, even when they otherwise mishandle non-ASCII:

Versions of jBCrypt before 0.3 suffered from a bug related to character encoding that substantially reduced the entropy of hashed passwords containing non US-ASCII characters. An incorrect encoding step transparently replaced such characters by '?' prior to hashing. In the worst case of a password consisting solely of non-US-ASCII characters, this would cause its hash to be equivalent to all other such passwords of the same length.

  • 1
    $\begingroup$ You can also consider en.wikipedia.org/wiki/Ascii85 for better compression while still being just ascii. $\endgroup$ – Aman Grewal May 24 at 19:46
  • $\begingroup$ Hi! Thank you for sharing the potential vulnerability for utf-8 encoding! I was trying to find the encoding type that handles almost all input cases (including non-latin, cyrilic, chinese/kanji, accented, etc) cases: unicode.org/faq/han_cjk.html. I should look into this too :) Cheers! $\endgroup$ – Benji Tan May 24 at 19:50
  • $\begingroup$ The reason for hashing prior to passing it to bcrypt is because bcrypt is unable to receive >72 bytes for its password. Using sha256 would ensure a 32b digest that could work (sha512 gave me the problem i encountered above), thus uniformly fixing my implemented password encryption pipeline! $\endgroup$ – Benji Tan May 24 at 19:53
  • $\begingroup$ @Benji Tan: SHA-256 alone won't fix your problem. SHA-256->lowercase hex->bcrypt would. The question has SHA3-512, otherwise I'd have suggested that. $\endgroup$ – fgrieu May 24 at 20:37

bcrypt is still considered a reliable password hash (when used with an appropriate cost parameter for modern hardware) but it's quite old and has a bunch of odd quirks; the 72-byte input limit is one of them. As discussed in the other answers, what PyCA's bindings are doing to work around this limit is cryptographically dubious. I would suggest you replace bcrypt with argon2, which is a current-generation password hash that does not have any of these quirks.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.