I know that the private and public key in assymetric cryptography are different and the public is used for encryption while the private for decryption. My question is if they are symmetrical to each other in a mathematical way? They must be because the one is inverting what is done by the other.
The public key and private key are related, but that does not mean that they have to be symmetric, and normally they are not. For instance with a public/private key in RSA, the public exponent is normally a small value, while the private exponent has a larger value. They can have the same properties though - making them symmetric in that sense - but this is not common.
As for EC public and private keys, these have different properties. One is actually a vector and the other a point. So there is no symmetry there.
What is common is that the public and private key share some properties. For RSA keys this is the modulus, and with EC keys the domain parameters should be equal for the algorithms to work.
I don't think it is wise to think of a public and private key to be symmetric, even though both keys are part of the same mathematical framework.