I'm under the impression that GPUs are only used to optimize some (not all) of a given cryptographic operation. Furthermore, I've only seen this GPU optimization done in Bitcoin.
This observation unleashed a number of rhetorical questions that relate closely to each other.
Rhetorical questions (answer if you like)
Outside of Bitcoin, are general purpose cryptography operations offloaded to GPUs in general when native CPU instructions don't exist (CMUL/PCLMULQDQ)?
Which cryptographic operations? (BigInt^BigInt? * BigInt)?
Is only some of the cryptographic work offloaded, or is all of the work? E.g. an entire AES/PGP runtime embedded in the GPU?
What prevents security risks? E.g. WebGL might be isolated from other processes.
Is the physical RAM in a GPU different? Exploits in the past describe how neighboring CPU banks can have their value inferred by the active process. Not to mention physical shielding of the RAM.
If any of the above could cause a security risk, is there any benefit with a platform that restricts access to the GPU? For example, the Metal graphics API is constrained in function and doesn't expose a "raw" GPU mode. (I think)
As a hobbyist, should I explore porting cryptographic code to Apple's GPU? This is the Bouncy Castle function that is slowing me down: StackOverflow -- How can I improve performance of BigInteger as compiled to Objective C by Xamarin.iOS / Mono?
What would a professional do to make that effort worthwhile and more secure?