It is not physically impossible to retrieve data, merely very difficult. Physical sensors such as light sensors and mesh layers on chips can be bypassed.
Christopher Tarnovsky's work using a Focused Ion Beam (FIB) electron microscope is worth a watch[1]worth a watch. IIRC the FIB he's using cost him somewhere around \$500,000 used, over 10 years ago. Renting time on one is about \$400/hour. It takes months of work to figure out how to hack a given TPM/HSM, and destroys numerous sample devices. That's ignoring the costs of the time of a skilled semiconductor engineer to conduct the attack. It's very, very difficult, but not impossible.