The goals stated can't be met according to the requirements defining traditional cryptography (including, public-key cryptography); most importantly, that all methods used except secret keys are public, and communication channels can be eavesdropped and altered (as routinely occurring on a pwned computer, or one fully under control of its rogue user).
The current question remains unclear on if the client computer(s) contain some initial secret/private key (and a certificate, if that secret is not initially also known to the server). I assume they don't. If they did, and could be trusted, and operated by trusted parties, the problem becomes trivial (see also butlast paragraph of this answer).
Goal 1
If the software is public, and there is no initial secret on the client side, then from a cryptographic standpoint, the only goal SSL/TLS (or cryptography) can reach is confidentiality of messages to the server, and authenticity+integrity of messages from the server.
For other directions: we face the problem that without an initial secret tied to a determined entity, we can't make a secret tied with a determined entity, thus can not encipher to or determine origin from a determined entity, in the sense these things have in cryptography; among the best that can be obtained:
- A reverse-engineering of the software is required (and hard to some varying degree), in order to compromise the confidentiality of messages from the server that the software does not otherwise output in clear, or to compromise the authenticity+integrity of messages generated by the software itself (rather than accepted as input); otherwise said, the software is considered obscure enough to hold some secret making it a cryptographically identifiable entity.
- On a trusted computer (thus excluding one pwned or under full control of a rogue user), after first running an unmodified copy of the software performing an appropriate protocol (that could be a key exchange protocol, or a certification request protocol), this computer/software can hold a secret and become a determined entity, which messages can be authenticated+integrity checked, and which can receive confidential messages that no other computer beyond the server can decipher. Notice that the server can practically not distinguish between this desired situation, and a different reality where the secret ends up shared among many rogue users (for example, the software can be run in a single VM, and the usage of that VM allowed to several rogue users one at a time, including with that VM running on their computer, and the server seeing a single IP thanks to a VPN; see also last paragraph of this answer).
Goal 2
If the software is public, and you assume that the original machine is operated (or pwned) by an adversary (including remotely), then from a cryptographic standpoint, there is no way to detect that a copy of the software is running. Among the best that can be obtained:
- It is detected/prevented simultaneously running unmodified copies of the software on two machines.
- It is detected/prevented alternatively running unmodified copies of the software on two machines that are not both cooperating and communicating with each other.
- Reverse-engineering of the software is required (and hard to some degree) in order to run, unknown to the server, a modified copy of the software that is largely equivalent to the original program, except for functionalities that the server performs and local software cannot; in addition, the required reverse-engineering can be made cryptographically impossible (by encrypting the corresponding code) for functionalities that have not been observed running at least once on an instrumented authorized software.
The method in the question seems to be able to meet 1 and 2 for goal 2. But
- The devil is in the details; in particular, the system should work reliably in case of message loss in the "server will send some random number to client" step, which can happens by accident, or be induced in many ways external to the software (as in issue #1 in this answer to a related question). Thus the server will sometime have to tolerate two distinct values in the next interaction with the client. It is hard to get that working reliably and not open to fraud allowing simultaneous use of two copies, especially by an adversary controlling both computers.
- In real life, there's software cracking, thus assumptions made that the software is unmodified are unrealistic.
- Pwned machines can cooperate and communicate with each other unknown to their respective honest users.
- Rogue users of any machine they fully control can make these cooperate and communicate, and even pretend being pwned as an excuse if caught.
If "ignoring software cracking etc." for some wide definition of that (which can match some real-life situations where there is no competent and determined adversary with access to a working instance of the software); and, for goal 1 in both directions, an initial secret unique to each client, and usable for authentication of the client (e.g. because it is the private key to a public key system, and the corresponding public key is certified, and the private key is in trusted hardware like a TPM); and tweaking the protocol in the question as explained; then yes the protocol outlined in the question can achieve goals 1 and 2, at least within the limits I stated.
People who never wrote an hello world! program in any compiled language can install software in the free Virtualbox software in order to circumvent software copy protection. That would be applicable even if the software allowed use only on a machine with a valid SSL certificate and matching private key (which would be highly unusual, and inconvenient to normal users). The technique in the question will not prevent these people from putting that VM on a USB drive that they can move around or lend, or "put in the cloud" with some caching making it quite usable. The technique in the question will prevent multiple instances of that VM to run, and thus discourage the original owner from allowing multiple instances, for fear of being unable to use the software. In that sense, the technique is practically useful. However this is not crypto, which is on topic on CSE; this is security by obscurity, which is not.
