Does AES-GCM without security tag turn into regular CTR Mode?

Question

We are looking to implement a simple password based key agreement scheme. Part of this is deriving a symmetric key from a common passphrase known to both parties involved.

This symmetric key is then used to encrypt an ephemeral public key A and the resulting ciphertext is then sent to the other communication partner who does the same with his public key (B). Both sides then calculate the common secret using a regular Diffie-Hellman approach (both sides also include two additional static keys)

Both sides then complete a key confirmation step in which they assert that the common secret is indeed common and no man in the middle is between them.

Since the public keys A and B are randomized, an attacker should not be able to run a brute force attack on the password used to derive the symmetric key since an attacker should not be able to determine whether he has succesfully guessed the correct password as the output of the decryption function should also be random no matter what key is used to decrypt the random public keys.

Is this a safe assumption to make?

Now to the question stated in the subject line:

There is already an implementation of an AES-GCM mode as authenticated encryption scheme in existence to ensure integrity and confidentiality of exchanged ciphertexts. As stated in the previous paragraph this mode of operation would allow an attacker to have access to a sort of oracle to start an offline attack on the password used during the encryption process of the public keys and is therefore not suitable for that specific task.

The question therefore is:

If we use the implemented AES-GCM mode but without security tag (TAG_SIZE 0 bit) does this make the cipher behave like regular CTR mode as to the attacker can no longer start guessing passwords and knowing whether he has guessed the correct one and thereby losing this potential oracle?

/edit: To clarify: We plan to implement EAP-EKE as key exchange protocol (https://tools.ietf.org/html/rfc6124) I thought this to be not essential to the question in the title. In my understanding the protocol relies on the described behaviour (not being able to distinguish a correct password guess from an incorrect one) to guard against dictionary attacks on the password.

Answer 1

I'll keep to the final question as security evaluating the entire scheme would be off topic:

If we use the implemented AES-GCM mode but without security tag (TAG_SIZE 0 bit) does this make the cipher behave like regular CTR mode as to the attacker can no longer start guessing passwords and knowing whether he has guessed the correct one and thereby losing this potential oracle?

Yes, GCM would then just get you CTR mode, but with a specific way of calculating the initial counter: $n << 32 + 2$ where $n$ is the encoded nonce, shifted left 32 bits (multiplied with $2^{32}$), and then added the value 2 in the zeroed, least significant bits. The proof is in the implementation.

---

Notes:

• The implementation I pointed at uses a (GCM default) nonce of 12 bytes; if you do not do that then calculating the initial counter will get more complex.
• Not all libraries will allow a tag size of zero as that obviously falls outside the acceptable range for the size of the authentication tag $t$.

---

Off topic, some remarks on using GCM in your protocol:

As others have commented, creating (or in this case hacking) your own protocol is probably not the safest thing to do; there are already schemes out there that will do this for you. And just using CTR instead of GCM should be preferred if you go with this scheme.

Beware that the decryption will always succeed, but that parsing / handling the plaintext may still result in an oracle. E.g. if you encode a number and send it then decoding could possibly fail, resulting in an oracle.

Answer 2

Is this a safe assumption to make?

Not unless you use RAW public keys. If it's Elliptic-Curve DH, then you can only have the X coordinate without header byte; if it's Finite-Field DH, no formatting either.

If we use the implemented AES-GCM mode but without security tag (TAG_SIZE 0 bit) does this make the cipher behave like regular CTR mode as to the attacker can no longer start guessing passwords and knowing whether he has guessed the correct one and thereby losing this potential oracle?

Are you not going to authenticate post-handshake data? Adversary can attempt a guess each time, and take a piece of post-handshake (possibly application) data to confirm whether he's got the correct key. Please reconsider before you drop all althentication-integrity-related information and before using your in-house key-exchange protocol.