Per my understanding of these comments, the question's test takes $\ell=64$ bytes from a Windows RNGCryptoServiceProvider, makes it $n=2\ell=128$ groups of $b=4$ bits, counts the number $n_i$ of groups with each of the $2^b=16$ bit patterns $i$, computes $p_i=n_i/n$, and for a limit $x=0.9$ checks
The question's test should only fail very rarely. The probability that it does even once in one million tests (as reported) seems to be less than $0.2\%$ according to my simulation: I have seen $25$ events for $18\times10^9$ samples tested. Something is at odds, independently of what follows.
The test is inspired by the formula for Shannon entropy per bit of a source of $b$-bit independent values of known probability $p_i$, but here we do not compute a Shannon entropy, because we are operating on a sample, not on the source, and Shannon entropy is not defined for a sample. The $p_i$ are experimental frequency, not a probability for the source. I can't see how this formula can be justified.
Update: I have asked about that test there, and got a detailed answer. It turns out to be known as the G-test or Woolf's log likelihood ratio test, and a sound alternative to Pearson's $\chi^2$ test.
Can I use something like Shannon entropy to determine the suitability of a symmetric key?
If the source was known for certain to be from a construction aiming at being a CSPRNG, no. Tests on such output can only give a fallacious insurance of security, or detect gross errors. The correct method is to scrutinize the specification and if possible the implementation of the CSPRNG.
But that's not what the question is about.
I am worried that employing such a filter to protect against human error may in turn create a security risk.
It does not in practice. The two following nitpicks are theoretical only:
- Restricting to keys that pass the test obviously reduces the keyspace to some (imperceptible) degree.
- Performing the test may leak information about the key, if side channel attacks are an issue. However, if they are, that's the problem, not the test.
I do have serious reservations on trusting that test to catch a bad CSPRNG:
- The most common issue with CSPRNGs is improper seeding, and the test won't catch it.
- More generally it is demonstrably impossible to prove or even meaningfully confirm the quality of a CSPRNG from its output: we can make CSPRNGs that pass any pre-existing test, yet are extremely weak for who knows their internals.
Independently, as already stated, if only a million tests have reported a fail (and the key is 64 bytes as stated), that's most likely an anomaly. It could be in the test, or in how the generator is used, or in the assumptions on what the generator is designed to output, rather than in the generator itself.
A poorly chosen key was installed into the system by mistake
The test indeed has some chance to catch some forms of that, and if the small false-positive rate (much less than one in 100 million keys if I get it correctly) is tolerable, why not? However, attempts to detect errors from the value of the key itself are bound to be unreliable. For example, the test won't catch a favorite test key for DES,
A preferable practice is to have procedures that prevent such mistakes. Like, transmitting keys using dedicated devices, or data blobs with integrity protection, or as a last resort well-thought Key Check Values.