Using the hash of the file $B$ to create an encryption key is a simple way to prove knowledge of its contents before providing decryption access to the file $A$, but there are some important details to consider.
Issues:
- Say someone has access to a large portion of $B$ but not the whole thing. The entirety of $B$ may be unguessable, but if some of $B$ is known, the unknown portion may be short or predictable enough to be guessed. If so, a correct guess could be confirmed by seeing if a key derived from the known part of $B$ and a guess for the remaining part $G_i$ succeeds in decrypting the file $A$. It may be very efficient to make guesses $K_i = H(B \space || \space G_i)$ which then lead to a sensible decryption result. This would violate your requirement that no part of $B$ can be recovered from the encrypted $A$.
- The use of a plain hash function, instead of a construction like HMAC, would leave the system open to known vulnerabilities. So let's assume from this point forward we'll use something like HMAC. Relying on the unguessability of $B$ for the confidentiality of $A$ seems dangerous, since there's no guarantee that an arbitrary file will by itself contain an adequate amount of entropy to derive key material.
- There should be some context information, and preprocessing of inputs, used along with a hash function to prevent issues where derived keys are reusable in illegitimate contexts.
Potential Solutions:
Require someone in possession of $B$ (a prover) to prove complete knowledge of $B$ before you (a verifier) give access to information $C$ which is used to derive the encryption key $K_A$ for file $A$. Below is one possible way to do this:
$K_A = HMAC('ContextForEncryptionKeyForFileA' \space || \space B, \space key=C)$
$W_{prover} = HMAC('ContextForProverChallengeBA' \space || \space T_{prover} \space || \space I_{prover} \space || \space B, \space key=R_{prover})$
The prover sends the verifier a timestamp $T_{prover}$ with adequate granularity, a random 32-byte token $R_{prover}$, optionally some identity information $I_{prover}$ like a username or even a public key, and the hash result $W_{prover}$.
The verifier can then check to see if the timestamp is valid: $T_{verifier} > T_{prover}$; and that it was created recent enough: $T_{verifier} - T_{prover} < Threshold$. Then it may be possible for the verifier to ensure the prover has adequate permission to request decryption access to $A$ using the identity information $I_{prover}$. If everything checks out, the verifier then checks to see if $W_{prover} = W_{verifier}$ using a timing-safe comparison and its own knowledge of $B$.
If the verifier's results are the same as the prover's, then the prover is given the key $C$, which could be 32 random bytes. If not, then the connection is terminated, and the verifier could keep track of how many failed attempts for decryption access to file $A$ were initiated by the prover associated with identity $I_{prover}$.
Final Comments:
This is an incomplete overview of a potential protocol which you could use to solve your problem.
Very importantly, it's missing the specifics for how the communication channels between prover and verifier are secured and authenticated. Similarly, if $I_{prover}$ is used to implement access controls, it would be important to also verify the prover is the legitimate entity associated with that identity.
In regards to the use of hash functions and $HMAC$. It's prudent to not just concatenate inputs together, but to encode them prior to hashing so as to mitigate canonicalization attacks. One simple way this is done is to count the number of inputs, measure the length of each input in order, then concatenate the inputs and prepend the metadata you just calculated. This helps ensure no potentially exploitable, or error inducing, unintended significance can be attributed to the inputs.
