I did some more reading. It would appear that the way a key is generated varies from system to system, so a lot of it would depend on the method used to generate the keys.
From what I see it is common to use the passphrase to encrypt the private key, and use a CSPRNG (cryptographically Secure Psudo-Random Number Generator) to generate those keys. So in those scenarios the passphrase encrypted private key is never published. Even if it were (bad idea) there would still be no way to compare the two and see they used the same passphrase or determine the passphrase though having both.
Because an RSA public key is (basically) the product of two prime numbers, the key cannot be directly generated from a passphrase. For example the passphrase FOO does not directly translate into two prime numbers. Instead that passphrase can be used as a seed, or part of a seed for a system that generates prime numbers (usually combined with a random seed or system entropy since a passphrase does not contain enough entropy to generate even an 128-bit key)
Even if the system only used a user provided passkey to generate the prime numbers, the prime numbers would be so totally different that were used for the two different key sizes, and they would not follow any deterministic pattern, so there would be no relation between the two keys.
For example: Say the passphrase FOO always generates the 4-bit primes of 7 & 13 (for a product of 91) when it passes through the 4-bit algorithm. If that same passphrase was used in the 8-bit prime algorithm it could generate 10007 (not saying it would, but as an example of a possible relationship) as the first prime, but 100013 is not prime, so it would need to generate 10037 (for a product of 100440259) or some other prime. So even if the passphrase was used to directly generate a number, there is no guarantee that the generated number would be prime, so additional steps must be included to find a prime. These steps decouple the relationship between the generated key and the passphrase / seed used to generate it.
A good simple explanation of RSA.