Yes, the scheme is weak, and made weaker by adding the SHA-512 hash of the password.
The ciphertext being assumed known, this scheme allows testing if a user password is genuine with little effort:
- compute the SHA-256 hash of the password to be tested
- using that as the key, decipher at least the portion of the ciphertext corresponding to the SHA-512 hash of the password
- compute the SHA-512 hash of the password to be tested, and compare it to the above. If it matches, the password guess is most likely correct (is not, it is certainly wrong).
There is no attempt to make any of this process slow. Therefore the scheme is awfully vulnerable to a password cracker such as these, which will simply test likely passwords using this procedure (or, most likely, a simpler version thereof using another redundancy in the plaintext, but we can't tell which from the question). With the exponential progress of CPU power, it is safe to say that most passwords humans will use are vulnerable to password cracking in such a scheme; this trend will only get worse with hardware progress.
Main problem is: building a key from a password is the job for a properly parameterized Password-Based Key Derivation Function, such as scrypt. As an aside, passwords must be salted. Failures in this regard are common mistakes in amateur encryption programs (and a number of others).
No, the scheme does not insure message integrity, and adding the SHA-512 in the plaintext does not help towards that.
The adversary can simply alter another portion of the ciphertext, and with many common modes (CBC, OFB, CTR), or if the SHA-512 hash of the password is at the beginning of the ciphertext, this will not prevent decryption. Plaintext might be garbled, but this still goes against the definition of message integrity.
Further, with many common modes (OFB, CTR, and CFB for the last block of the file), if the adversary knows some section of the plaintext (e.g.
TWO), she can change it to anything of the same length that she wishes (e.g. to
For encryption and authentication, use an authenticated encryption mode, such as AES-GCM. Or a more traditional MAC, using a different key than the one used for encryption, especially if this MAC is based on the same block cipher as used in the encryption. In password-based encryption with integrity protection, that different integrity key can be derived from the same password as for encryption, but a different salt.
And of course, if the adversary succeeds in finding the correct password, she can encipher any message she wishes even with the best such schemes.