There are several reasons for an authenticated decryption (with AES-GCM or any other AE or AEAD mechanism) not to return any plaintext if the ciphertext is not authentic (i.e if the tag does not match).
One danger is if the calling code starts using the partially decrypted plaintext. Suppose the caller does something with the beginning of the plaintext, then learns that the ciphertext was not authentic. The caller must then undo whatever was done, since this could be entirely spurious. Undoing what has been done is not always possible: if you've already sent a message over the network, you can't unsend it. If you've created a file, you can remove it, but traces of the data may remain in storage.
Furthermore, the way in which the code reacted to the partially decrypted plaintext may have indirect characteristics that are useful to an attacker. Side channels in the response to the partially decrypted plaintext may reveal parts of this plaintext to an external observer. This can happen with authentic plaintext, of course, but the situation with non-authentic plaintext is worse. If the plaintext is authentic, you know that it was created by a trusted party, and that usually means that it is well-formed in some sense. If the plaintext is not authentic, it may be ill-formed: it may contain numbers that are out of range, strings with invalid content, etc.
There are further problems if the entity that requests the decryption is doing it on behalf of a client. That is, suppose that Alice processes messages from clients, and the first thing the client does is submit an access token which is protected by (for example) AES-GCM. The first thing Alice does is to decrypt the token, and she reads that the token contains Bob's identity, so she starts reading the information she has on Bob and revealing some of it to the client (because after all why would you prevent Bob from retrieving information about himself?). But the token was actually produced by Eve, who can't create a valid token for Bob (because she doesn't have the key shared between Bob and Alice). And so Alice started to reveal Bob's information to Eve, which she will realize once she gets to the end of the ciphertext and learns that the token was not authentic.
To avoid these problems, you must not start processing the content of an authenticated-encrypted message until you have verified its authenticity. Ideally, this is enforced at the level of a cryptographic API, which either returns the authentic plaintext or a failure code. Sometimes, this is not possible, for example on an embedded system that must decrypt a message that doesn't fit in RAM. In such cases, the layer that processes the partially decrypted plaintext must not look inside the content: all it may do is store it temporarily in a secure place. Effectively, that layer becomes part of the cryptographic processing code, and it must take care not to leak unauthenticated plaintext to its caller.
If the job of the intermediate layer is to reveal the plaintext to an untrusted caller (for example, if you're writing code in a secure enclave), then the intermediate layer must take care not to reveal any unauthentic plaintext. If you do reveal unauthentic plaintext, you're allowing your caller to decrypt arbitrary data, and not just the messages that it may be entitled to.
If your intermediate layer doesn't have enough memory to store the not-yet-authenticated decrypted plaintext, one possible trick is to encrypt it on the fly with an unauthenticated cipher, then have it decrypted.
- Generate a single-use key K₁ for an unauthenticated cipher, for example AES-CTR.
- For each chunk of the ciphertext:
- Perform a chunk of authenticated decryption, putting the result in secure memory. (By “secure memory”, I mean memory that belongs to your intermediate layer, and that is not shared with your untrusted caller.)
- Encrypt the result with the stream cipher, putting the result in shared memory.
- Verify the authenticity of the authenticated ciphertext.
- If it isn't authentic, erase K₁. The untrusted caller has data encrypted with K₁ but does not have K₁, so it can't do anything with that data.
- If it is authentic, reveal K₁ to the caller. In practice, you might as well do the decryption (this is often preferable for performance since your layer is typically closer to any possible hardware acceleration, but sometimes the situation is the other way round if your code is running in a secure enclave which has less good performance than the untrusted caller). The caller may have modified the shared memory, but doing the decryption of modified data doesn't give the caller any advantage, since we're willing to reveal the key anyway.