The answer to the question in the title is: Yes, it is possible to use RSA signature for software licensing (e.g. in the context either RSASSA-PKCS1-V1_5 or RSASSA-PSS in PKCS#1 as defined here, or any of the three schemes in ISO/IEC 9796-2 free but partial preview). One minor issue is that, for a decent security level, what protects the license file will be much too big to be keyed in.
There are a lot of terminology errors in the body of the question:
- What's wanted is distribute a
crypted encrypted signed license file to many customers
- What's needed is use the RSA
public private key to encrypt sign that license file; in signature schemes with message recovery (e.g. ISO/IEC 9796-2) this will have the (perhaps desirable) side effect of making at least some of the signed content not directly intelligible, though recoverable without any secret information;
- The software that will be licensed to customers will have inside the
private public key in order to be able to decrypt verify the license content (and recover what portion of the signed information was not directly intelligible, if applicable)
- The question is if it can be possible to
know obtain the public private key giving given the encrypted signed file, the private public key and the final plain text signed data.
The short answer to that other question is: No, there is not a reason to worry that the private key (or anything allowing to generate new signed license files) could be obtained, provided the legitimate private key is kept and used securely, and was adequately drawn.
You'll get one private key to sign (and perhaps, with message recovery, make not directly intelligible some or all of) the data forming the license file, and an associated public key to verify the integrity (and perhaps, with message recovery, recover all) of that data; and where it will not be possible to obtain the private key (or some equivalent mean) given the public key and the data in any of its form.
On the other hand, no amount of cryptography will really prevent that someone skilled modify the software to bypass the license check, or allow the use of a license duplicated from an authorized one even if the software attempts to lock a license to a particular computer. Without temper-resistant hardware, the best you could do is that some portion of the software is unusable without access, at least temporarily, to a valid license to use that portion of the software.