The normal way is to keep secret data in volatile static RAM (i.e. 6 transistors per bit, like in current CPU caches, not with capacitors like in DRAM). When the power is cut, the data is gone in a matter of microseconds. The HSM contains a small battery to keep the RAM up and running for up to a few weeks. The battery also powers an array of sensors, e.g. on temperature, light, physical integrity of the covering and so on. If anything goes amiss, the system automatically shuts down the power, and the keys are gone.
So the trick, really, is not to store the data "in an hard drive, permanently".
An amusing consequence is that if an HSM is unpowered for too much time, the keys can be lost. Pro-tip: when you ship an HSM to a factory in China, label it as "a network card". Otherwise, customs will keep it around for a month, and the keys will be lost.
The above is valid for HSM and payment terminals. Smart cards, on the other hand, employ other tricks, which are undocumented (and most of them are trade secrets).