A lot of answers have said "a mathematical structure" which is absolutely right, but I still see there might be a question: how on earth can one exist? I'll try and fill that gap at a simpler level.
So a simple caeser-shift cipher might look like this: $x+7$. In our really simplistic, dangerous example, the "key" here is $7$. If I have some encrypted text, I know that by subtracting $7$, I get back to $x$. Therefore, this is a really simple example of a symmetric cipher, because what I do one way also allows me to undo it.
As you might have noticed, most domains in cryptography are finite - for example, there are only 26 letters in the alphabet. At some point you need to "wrap around". Mathematics provides us with a technique to do this called modulo arithmetic. Essentially, under a modulus, you can think "if I divide this by the modulus, the number I have is the remainder". some examples:
$4 = 4 \mod 7$
$8 = 1 \mod 7$ (8/7 = 1 remainder 1)
$4+7 = 4 \mod 7$ (11/7 = 1 remainder 4)
$-3 = 4 \mod 7$ (not so hard... what happens when you add 7 to -3?)
As you can see, arithmetic holds under modulo. Undergraduate mathematics courses rigorously establish these truths and if you're interested, read up on Number and Group theory. The next step is to understand that multiplication also holds:
- $2*4 = 1 \mod 7$ (2*4 = 8, as above)
- $5*3 = 1 \mod 7$ (5*3 = 15, 15/7 = 2 remainder 1)
and so on. Multiplication in mathematics often throws up some problems when it comes to inverses - for example, how do I go from 1 back to 5? I can multiply by 5. How do I go back to 2 from 1? Multiply by 2. These examples are not quite right in terms of what I wanted to show, so here are some more:
- $6*3 = 4 \mod 7$ (6*3=18. 18/7 = 2 remainder 4)
- $4*5 = 6 \mod 7$ (4*5=20. 20/7 = 2 remainder 6).
With these examples, I've shown that you can go from 6 to 4 and back to 6 using multiplication, but by multiplying by different things.
This is one, very simple way to create such a mathematical structure. The genius of RSA is choosing the numbers involved in such a way that one can easily determine how to get to an encrypted value, but not back again. I've explained it fully in another answer; however, in essence, it is simply a more complicated version of what we've done here. The clever part is understanding those structures and which choices of numbers make good/bad keys and which work/do not.
But you're telling me multiplication is hard to undo: what about division?
Firstly, it really depends on circumstances. Under some circumstances, like the trivial example I present above, finding an inverse or even using division is easy. It's important to think about what division means. In a rational (any number you can write as a fraction) field, multiplicative inverses exist in the form $p * (1/p)$ (as well as $p*q = 1$).
However, when considering RSA, note that encryption is $t\times t\times \ldots = t^e = c \mod n$ for some public key $e$. So to compute the inverse, we'd need to compute $c \times (1/t) \times (1/t) \times \ldots = c \times (1/t)^e \mod n$. The reason for this is that each multiplication of $t$ needs to be undone by an inverse $(1/t)$ but it should be clear that if we only have the ciphertext $c$, we don't know $t$ to compute $1/t$.
So our next possible route is to compute $c^{1/e}$ as $t^{e*1/e} = t$. This is equivalent to computing the eth-square root of $c$ (for example, $x^{1/2} = \sqrt{x}$) which is hard to do when under a modulus of the size that RSA requires you use - under certain circumstances. Under others, it's known as the "cube root" attack: see this presentation and this one.
Other public key crypto systems use similar observations - for example, Diffie-Hellman relies on this property:
$$a^x = b \mod n$$
Under certain cases of n (for example $(\mathbb{Z}_p, \times )$ i.e. when n is prime and we are interested only in multiplication and therefore a is greater or equal to 1) this is hard to reverse. This forms the basis of a number of other public cryptosystems.