but the key is stored physically anyway in the hardware, so access is still possible
Yes, in the end the value of the key is required to perform the calculations. And storing digital data still requires a carrier.
However, various measures can be taken to make sure that only the right users can use the keys, and that nobody is able to retrieve the key values.
This means that often HSM's are protected against power fluctuations and movement. Side channel attacks are often protected against. Sensors will detect when you are trying to open the outer shell. Disallowign accesss to the inner workings of an HSM is the property of being tamper-proof.
Not all HSM's are created equal. Some low end HSM's are just Smart Card-type chips with a USB connection, e.g. Yubikey HSM's are those kind of crypto tokens. Of course, that also means that the storage capacity and speed are rather different from network or PCI based cards. It could also mean that some operations such as key pair generation are not as secure as their larger siblings.
Note that there are also software / cloud based HSM's nowadays that rely on homomorphic operations accross heterogenous environments. In that case the location of the key is distributed over various devices.
if the keys are encrypted, that means that there is another key in the hardware that encrypted those keys. this key is not encrypted then !!
It is not said that keys in a HSM are encrypted, but generally they probabably will be, if just because destroying them would be that much easier.
In the end the key needs to exist. You cannot have "turtles all the way". However, the final key or keys may be stored into specially designed silicon, making it near impossible to retrieve the final key value.
And as ciphers are considered pretty strong, many security devices even store their keys outside the device, loading the keys into the device only when they are required. That way the key storage is unlimited and the key value and operations using the key are still protected against leakage. This is the general way that a TPM works, for instance.