There is definitely a mistake being made somewhere here. I believe you have fundamentally misunderstood how the hill cipher works.
- Your plaintext should be a string of characters, not a matrix.
- The hill cipher key must be a square matrix, thus cannot be 3x4. The reason for this is as you pointed out, a non-square matrix does not have an inverse thus the ciphertext would not be decryptable.
- The hill cipher operates on chunks ("blocks") of the plaintext to encrypt.
No matter what your text is, you should be able to split it into encryptable/decryptable chunks which map between plain and cipher text. The length of each chunk, n, tells you the dimensions of your key matrix (n x n).