It's rather hard to answer the full question, but I'll try and answer as best as possible.
You try to derive a data encryption key using the IV, because you think the IV may be overused.
NIST SP 800-38D section 8.3 states (for a probability of non-repeation of $2^{-32}$ ):
The total number of invocations of the authenticated encryption function shall not exceed
$2^{32}$, including all IV lengths and all instances of the authenticated encryption function with
the given key.
Great, so unless you invoke GCM more than 4 billion times (short-scale), then you should be OK, and you may not need any special constructs.
Let's assume you don't want to limit the amount of GCM calls. In that case you to derive a key for each encryption. You can use a Key Based Key Derivation Function or KBKDF. Probably the best one you can use is HKDF. HKDF accepts a salt as well, and is able to generate as many bits from the input as needed. So you could use the following construct:
- $salt = random()$
- $seed = PBKDF2(salt, password, 256000, SHA2-256)$
- $key = HKDF(seed, \text{"key"})$
- $iv = HKDF(seed, \text{"iv"})$
- $(ct, tag) = E_{AES-GCM-256}(key, iv, data)$
- $output = encode(salt, (ct, tag))$
now say you don't want to perform those 256000 iterations for each file, and you want to encrypt/decrypt multiple files at a time, then you can do the following:
- $salt = random()$
- $seed = PBKDF2(salt, password, 256000, SHA2-256)$
- $salt_2 = random()$
- $key = HKDF(seed, salt_2, \text{"key"})$
- $iv = HKDF(seed, salt_2, \text{"iv"})$
- $(ct, tag) = E_{AES-GCM-256}(key, iv, data)$
- $output = encode(salt, salt_2, (ct, tag))$
as HKDF can also use a salt. For each file, repeat steps 3 to 6.
A trick is to use just HKDF-expand and generate 512 bits for key and IV. Other, more simple KDF's such as KDF1 or KDF2 may also be used if HKDF is not available.
As for your scheme, I see some drawbacks.
- the additional call to SHA-3 is spurious;
- the IV of GCM is best set to 96 bits (12 bytes), but the XOR and encryption of the IV seems to suggest an IV of 128 bits;
- the XOR'ing of key and IV is not a secure key derivation function;
- the key length is now limited to the size of the IV or introduce additional complexities;
- reusing the derived key for different, related algorithms is not recommended.
Here the combination of 3 and 5 is the biggest issue; you'd have to prove that the insecure key derivation doesn't hamper the security of the GCM algorithm. This is not something you should want to do.
As always, it may be advisable to use an existing protocol and library though; there are plenty secure ones available.