I am currently carrying scholar research about IPsec protocol, and I have a question regarding the IP Encapsulating Security Payload (ESP). Is the initialization vector of operation mode transmitted in clear in the message ESP Payload ? If it is transmitted in clear, why is it not a security issue ?
Yes, the initialization vector is transmitted in the clear as first part of the ESP payload data (see RFC 4303, section 2, and the RFCs that define the use of specific algorithm with ESP, e.g. RFC 3602, section 3 for AES-CBC, or RFC 4106, section 3 for AES-GCM).
The security of the encryption and integrity protection does not depend on the IV being kept secret. Its primary use is that of a nonce to ensure encrypting the same data with the same key does not result in the same ciphertext.
Some algorithms require that the IV is random/unpredictable (e.g. AES-CBC) but in many modern combined mode algorithms (e.g. AES-GCM or ChaCha20/Poly1305) it just must not repeat and can simply be a counter.
Under certain conditions it might not even be necessary to transmit the IV, for instance, because it can be derived from other information, like the sequence number in ESP. Therefore, there is currently an internet draft under discussion that proposes to omit the IV for certain algorithms (e.g. AES-GCM) and instead just use the (extended) sequence number, which is what's generally already used anyway, it's just transmitted twice in each message.