From my understanding no.

If you look at the [OpenPGP spec section 3.7][1] it covers the String-to-Key (S2K) Specifiers which convers a passphrase to a key. Assuming the Salted S2K or Iterated and Salted S2K method is used (which is recommended) then reusing your passphrase shouldn't be an issue. The larger key has a larger length, which means the passphrase will be concatenated more times with the salt. Good hash algorithms generate significantly different hashes for even small changes in the plain text. So the two generated keys should bare no similarities that could indicate what the passphrase may be.

Also hash algorithms are called one way algorithms, meaning you can't get back to the plaintext from the hash value. This means that you cannot get from the 2048-bit key back to the passphrase and then use that passphrase to break the 4096-bit key. 

**Update:** I was thinking about this some more. If the lower bitrate key used an insecure hash algorithm then it could be easier to reverse, thus providing the passphrase. So the above holds true if both keys are generated with the same hash algorithm.

  [1]: https://www.rfc-editor.org/rfc/rfc4880#section-3.7