Yes, if you have a 100% certainty that you can measure each timing then this is the way to go about it, approximately.
You'd have 128 tries on average per byte, so you'd need 128 * 4 = 512 tries, or $2^9$ bits of security to break, that's next to nothing. This is of course an average, the attacker may be lucky or unlucky, the maximum amount of tries would indeed be 256 * 4 = 1024 tries (or 1020 really, as you don't need to test the last byte value due to the power of deduction) and 1 in case of extreme luck.
There are however a few things wrong in your question:
timing attacks often require a large number of retries to find just one correct value - timing attacks usually rely on statistic, instead of an oracle that immediately tells you if you're correct or not.
you don't need to bother with the exponentiation of 2; there are just a maximum number of tries per byte: 256 (or 255), so the probability of guessing the first byte is just $1/256$, next guesses however will have a higher probability (as the number of untested values goes down).