What are the best practices for IV generation for AES and other algorithms?
Is RNG enough?
What if I cannot use RNG and need to use a variable that is related to some characteristic of the encrypted message? For example, file number or ObjectGUID? Will encryption of the IV do the job?
If encryption of the IV is a solution shall I use the same key that encrypts the actual message or a different key?
The requirements differ per mode of operation. AES itself is a block cipher, and as block cipher, it doesn't take an IV at all. Tweakable block ciphers may take a tweak, which may have some overlap with an IV, but AES isn't tweakable by itself.
CBC requires an unpredictable IV (to the adversary). One of the common ways is indeed to generate a 16 byte (one block) random IV. However, to do this you would require a cryptographically secure RNG, because a non-secure RNG may well be predictable given previous output.
You can use other, unique, known information as IV in some instances. However, CBC requires unpredictable data and a 16 byte IV. So e.g. a file number doesn't cut it by itself. However, you could use tricks to make it fit. For instance, you can use a secure hash on the unique data and use the leftmost 16 bytes as proto-IV. Then you can encrypt the proto-IV using the AES block cipher and use the result as IV. That way you would get an IV that is unpredictable and that doesn't have to be included with the message itself. Note that you should not perform decryption for that IV; the encrypted proto-IV is the actual IV.
Using a separate key would make your protocol ever so slightly more secure, as it brings down the chance of collisions (which an adversary might try to trigger deliberately, if possible) but it isn't required (in other words: you'd use a different key on PC's, but for embedded / lightweight systems you might a different trade off).
As indicated, creating a protocol is tricky and there may be many (other) attack vectors. Generally we try and use authenticated encryption schemes nowadays, as it at least doesn't allow for alteration of the encrypted message, which - for CBC - may also completely compromise confidentiality if plaintext or padding oracle attacks are feasible.
