The general limits from the NIST recommendation are as follows:
Maximum Encrypted Plaintext Size ≤ 239 - 256 bits;
Maximum Processed Additional Authenticated Data ≤ 264 - 1 bits;
This stack overflow answer (https://crypto.stackexchange.com/a/20340/44337) hints that the maximum invocation of 239 - 256 bits of encrypted plaintext is too big and “you should be advised to stay well away from those limits.” In this case what kind of limit makes sense where I can optimize the use of a generated key, but also not decrease the security of the cipher text? Is reaching the maximum invocations of the encryption operation of 232 operations considered too dangerous? Should I go ahead and limit the amount of invocations to 231? Or may be half that to 216? I have the ability to generate new keys at any time for any file. So the question is what is a good trade-off between maximum key use and security. Obviously I can’t process more than 239 - 256 bits with the same key.
Some of the comments on stack exchange mention the maximum number of 96 bit NONCEs to use with the same key is 228.9 which will keep the security above 264 (Is the length limit of AES-GCM per key or per (key, nonce) pair?). I am strictly trying to figure out what the maximum safe size of the processed message with the same key/NONCE is before the need to re-generate the key or change the NONCE.
Another stack exchange question deals directly with the size of the encrypted message but does not give a recommendation of the safe maximum size (Plain text size limits for AES-GCM mode just 64GB?). Instead it shows that the security depends on the tag and the number of encrypted message blocks. The relationship is n - k, where n is the tag length and k is the number of encrypted blocks. So for a 96 bit tag processing a message of size 232 blocks, this gives us 96 - 32 = 64 or 264 security. Which is pretty low. Other commenters say to chunk the encryption at around 10GB per chunk. But what kind of security are we achieving then?
It seems that AES-GCM is not well suited to encrypt large files unless more complexity is added in chunking the files appropriately at some safe limits such as 1 GB or 10 GB etc. May be a better solution would be to use ChaCha20/Poly1305, but that is out of the scope of this question. I want to see what the current limits of AES-GCM are before I decide to change the cipher.