can I determine the IV
In CBC block cipher mode of operation, not restricted for AES, the decryption of the first block is
$$P_1=Dec(K,C_1)\oplus \text{IV}$$
Where $P_1$ is the first plaintext and $C_1$ its encryption of $P_1$ with the key $K$ under the CBC mode of operation. Therefore
$$\text{IV} = P_1 \oplus Dec(K,C_1)$$
even if I don't know the padding
If you know the plaintext, there are some standard padding schemes that you can use to determine the IV, like
If the plaintext is more than one block, you don't need to consider the padding for calculation of the IV. The first plaintext block will be enough to calculate the IV.
The IV is only used as an initial XOR of the plain text in encryption
Shortly, yes. Longly; the CBC is a propagation mode, encryption process in CBC mode is performed as
\begin{align}
C_1 &= Enc_k(P_1 \oplus IV)\\
C_i &= Enc_k(P_i \oplus C_{i-1}),\;\; 1 < i \leq nb,
\end{align}
where $nb$ is the number of blocks. The IV is for the first block, and the rest encryption is using the previous ciphertext for the IV, chaining.
Decryption process in CBC mode is performed as
\begin{align}
P_1 =& Dec_k(C_1) \oplus IV\\
P_i =& Dec_k(C_i) \oplus C_{i-1},\;\; 1 < i \leq nb,
\end{align}
where $nb$ is the number of blocks.
Although the IV is used only in the first block, it affects all other blocks - propagation. We can see this better if we expand the equations for encryption
$$C_j = Enc_k(P_j \oplus Enc_k(P_{j-1} \oplus \cdots Enc_k(P_1 \oplus IV)\cdots)).$$