Yes, the idea of using RSA signature as KDF can work. The KDF's key will be the RSA private key, and the other input of the KDF will be the message to sign. Other requirements:
- Either:
- The signature is post-processed by a cryptographically secure PRF, e.g. a hash (which will make the output of the KDF indistinguishable from random, and the KDF uninvertible).
- Perhaps: It is kept only suitably little low-order bits of the raw output of the integer constituting the RSA signature. Without proof, I conjecture that for $n$-bit RSA, and secure deterministic signature padding, the low-order $n/3$ bits are indistinguishable from random, and the KDF uninvertible (including with the public key).
- It maters neither that the "secret material" output of the KDF is distinguishable from random, nor that the input of the KDF is reconstructible form its output (e.g. by one holding the public key, or because some bits are distinguishable from random).
- It is used a secure deterministic RSA signature scheme, including RSASSA-PKCS1-v1_5 of PKCS#1, and ISO/IEC 9796-2 scheme 1 and 3.
- The signing procedure does not force something variable in the signed message, nor in the part of the signature actually used.
Requirement 1 insures that the output of the KDF has the desired properties. With 1.1 or perhaps 1.2 (an unproven shortcut to save a hash), the output of the KDF is indistinguishable from random, and the KDF is not invertible. With 1.3 the signature itself can be the output (no matter how it is formatted, including e.g. base64), and that could be usable for a KDF generating, from public material, some output that only needs to be unpredictable and not colliding, but can be long and distinguishable from random.
Requirements 2/3 are there to insure that the KDF is worth the name function, that is the same input always yields the same output. Requirement 2 excludes randomized RSA signatures, including RSASSA-PSS of PKCS#1v2, and ISO/IEC 9796-2 scheme 2. Requirement 3 excludes, for example, a Smart Card that would increment an internal counter at each signature, and force that in the message being signed, or (combined with 1.1 or 1.3) as a suffix to the signature.
Unfortunately, I know no reference.