Security of this scheme

Question

I am developing a software and I am not sure if it is completely secure.

I have 22 Mb of data to encrypt used multiple times,
random IV (generated using RAND_bytes from OpenSSL lib)
and password which is $key = SHA3_{256}(PBKDF2(salt, password, 256000, SHA2-256))$.

SHA3_256 from Qt QCryptographicHash, PBKDF2 and SHA_256 from OpenSSL, salt is also generated from RAND_BYTE and 256 bits long.

Now I would like to know your opinion if this scheme is secure:

XOR key and IV together:

$$ encryptionKey = IV \oplus key\\ $$

encrypting IV:

$$ firstBlock=E_{AES-EBC-256}(key, IV) $$

encrypting data:

$$ secondBlock = E_{AES-GCM-256}(encryptionKey, IV, data) $$

and then:

$$ output << firstBlock\\ output << secondBlock $$

I am not sure especially about this part $E_{AES-GCM-256}(IV \oplus key, IV, data)$.

I would really appreciate your opinion about it.

Answer 1

It's rather hard to answer the full question, but I'll try and answer as best as possible.

---

You try to derive a data encryption key using the IV, because you think the IV may be overused.

NIST SP 800-38D section 8.3 states (for a probability of non-repeation of $2^{-32}$ ):

The total number of invocations of the authenticated encryption function shall not exceed $2^{32}$, including all IV lengths and all instances of the authenticated encryption function with the given key.

Great, so unless you invoke GCM more than 4 billion times (short-scale), then you should be OK, and you may not need any special constructs.

---

Let's assume you don't want to limit the amount of GCM calls. In that case you to derive a key for each encryption. You can use a Key Based Key Derivation Function or KBKDF. Probably the best one you can use is HKDF. HKDF accepts a salt as well, and is able to generate as many bits from the input as needed. So you could use the following construct:

1. $salt = random()$
2. $seed = PBKDF2(salt, password, 256000, SHA2-256)$
3. $key = HKDF(seed, \text{"key"})$
4. $iv = HKDF(seed, \text{"iv"})$
5. $(ct, tag) = E_{AES-GCM-256}(key, iv, data)$
6. $output = encode(salt, (ct, tag))$

now say you don't want to perform those 256000 iterations for each file, and you want to encrypt/decrypt multiple files at a time, then you can do the following:

1. $salt = random()$
2. $seed = PBKDF2(salt, password, 256000, SHA2-256)$
3. $salt_2 = random()$
4. $key = HKDF(seed, salt_2, \text{"key"})$
5. $iv = HKDF(seed, salt_2, \text{"iv"})$
6. $(ct, tag) = E_{AES-GCM-256}(key, iv, data)$
7. $output = encode(salt, salt_2, (ct, tag))$

as HKDF can also use a salt. For each file, repeat steps 3 to 6.

A trick is to use just HKDF-expand and generate 512 bits for key and IV. Other, more simple KDF's such as KDF1 or KDF2 may also be used if HKDF is not available.

---

As for your scheme, I see some drawbacks.

1. the additional call to SHA-3 is spurious;
2. the IV of GCM is best set to 96 bits (12 bytes), but the XOR and encryption of the IV seems to suggest an IV of 128 bits;
3. the XOR'ing of key and IV is not a secure key derivation function;
4. the key length is now limited to the size of the IV or introduce additional complexities;
5. reusing the derived key for different, related algorithms is not recommended.

Here the combination of 3 and 5 is the biggest issue; you'd have to prove that the insecure key derivation doesn't hamper the security of the GCM algorithm. This is not something you should want to do.

---

As always, it may be advisable to use an existing protocol and library though; there are plenty secure ones available.