AES-GCM internally directly uses the given key for AES-CTR mode, without performing any derivation. The initial counter for that mode relies only on the IV/nonce; given the recommended default of a 12 byte IV, the IV/nonce is used directly as well. So there is absolutely nothing preventing an attacker to decrypt the message when the attacker gets hold of the key. The additional data is required for calculating the right authentication tag, but it doesn't influence the decryption at all. Putting a password in the additional data does, for that reason, not ensure confidentiality of the message.
The authentication tag also relies on the same key for security. Although the attacker may not be able to verify correctness of the authentication tag, it would be possible to append data to the message undetected (decrypting the last GHASH result, then continuing the GHASH calculation and finally encrypting the result again - basically performing a length extension attack). The resulting plaintext may not match the expected result, but it depends on the protocol if it would generate an error and if that error indicates tampering. So putting a password in the additional data does, for these reasons, not provide integrity or authenticity of the message.
As for any symmetric cipher, losing the key is fatal. Using a password based key derivation function (a.k.a. password hash) such as Argon2 - as cipherfox correctly identifies - is best. Use the resulting value as key and another part of the result as IV. Even then, the strength of the password is fundamental to the security provided by the password hash.