In practice, “secrets do not leave the token” is not beneficial.
Ouch, I think I would rather like to know where my keys are. This is about the same as saying that you don't care if your house keys got copied or not.
For example, a banking application. This program is not able to execute on a token since tokens have insufficient UI. Hence this program will execute on a UI device to which the token is attached.
In PKCS#11 you can still have key specific PIN or passwords. This is especially important for transactions which is what you are referring to here.
Sometimes additional security measures such as counters are also present. If those are not depleted they can at least be used to check if the key was used unexpectedly. In software the attacker could just reset the counter at will. Cars also have odometer's that are deliberately hard to compromise.
That all said, yes, if an attacker has full access to the keys within the token over extended periods of time then the token is virtually useless.
If cryptographic algorithms are implemented in the token, the attacker can execute any operation that the user can as long as the token is attached.
Absolutely, and there is a balance between usability and security here, which is hard to maintain. E.g. you can setup a HSM in such a way that a person has to login into each session. Then again, you might have some trouble finding a human in the middle of the night that is able to do this upon a sudden restart of a server. This would be fine for personal tokens and smart cards.
Login procedures can also be protected by additional counter measures such as number of logins or delays between password verification checks.
If cryptographic algorithms are implemented in the UI device, the attacker posesses the user's secrets.
There are things like white box cryptography of course. And then there are certainly levels of access. Even in software you can have a secure system run at a different level than the user process. However, in that case you cannot easily protect against side channel attacks on processor usage. So this actually advocates for a hardware device such as a token.
AFAIK, PKCS #11 does not allow backups. If you store all your secrets in a token, it will be valuable not only to attackers, but to you too.
PKCS#11 is a generic token interface. However, most if not all HSM's have proprietary backup mechanisms and / or extensions of the PKCS#11 interface to deal with backups.
Other tokens such as smart cards basically require you to request a new smart card or token - which is fine for signature generation (authentication, non-repudiation). It is bit more tricky when it comes to decryption for obvious reasons.
Restricted set of cryptographic algorithms. Only algorithms implemented in a token.
Absolutely, and this can be a nuisance. And if you enable FIPS mode then only FIPS algorithms will be available. HSM's can receive (signed) updates though. And generally we don't use tokens for research, we tend to use them for standardized algorithms.
Some HSM's actually allow you to run restricted programs as well - although this may have interesting effects on certification of the hardware.
More complex hardware interface and API.
That's debatable. I think the PKCS#11 interface is well specified and it is about the only standardized, generic crypto API out there. It may not suite your language / runtime model perfectly, but the fact that you can use it from a myriad of languages and runtimes has a lot going for it. Many languages contain wrapper libraries around PKCS#11, which are then wrapped object oriented wrappers where required.