I know that the private and public key in asymmetric cryptography are different and the public is used for encryption while the private for decryption. My question is if they are symmetrical to each other in a mathematical way? They must be because the one is inverting what is done by the other.
The public key and private key are related, but that does not mean that they have to be symmetric, and normally they are not.
For instance with a public/private key in RSA, the public exponent is usually a small value, while the private exponent is a larger value.
As for EC private and public keys: these have different properties. The private key is a vector and the public key is a point (the base point multiplied by that vector). So there is no symmetry there in the first place.
What is common is that the public and private key share some properties. For RSA keys this is the modulus, and with EC keys the domain parameters should be equal; this is logical because both the private and public key both belong to the same mathematical framework.