I found a scheme for white-box RSA. It seems to protect the input and output of modular operations.
I'm curious about how to analyze the security of this solution.
Does anybody know anything about it?
That is a weird patent. The title says "RSA"; however the claims are strictly for elliptic curve operations (and while the description is more general, it is written at such a high level that it's hard at times to see what it's actually trying to say).
The ideas in the claims appear to be mostly blinding the ECC projective coordinates; this idea was published by Coron back in 1999.
In the end, the ideas behind the "white-box" parts of this idea appear to depend on:
You having a compiler that obfuscates the output, that is, "the method of obscuring software code is carried out by a compiler". That strikes me as a cheat - if you had a compiler that generated white-box-quality output, why do you need this idea?
An assumption that "none of the splitting secret multiplicative values and the secret splitting additive value are observable to an attacker"; that appears to be inconsistent with the assumptions of "white-box", where the attacker can observe everything.
Color me unimpressed (and it's hard to analyze the security, as it depends so crucially on the above two assumptions)