Transport mode is allowed between two end hosts only; tunnel mode is required when at least one of the endpoints is a "security gateway" (intermediate system that implements IPsec functionality, e.g., a router.) But in case all addresses are public and scalability is not an issue, why transport mode IPsec SAs cannot be used between two security gateways ?
That's because IPsec transport mode reuses the same IP header as both the 'outer header' (that is, the IP header that appears in front on the encrypted IPsec packet), and the 'inner header' (that is, the IP header that's the plaintext packet).
Suppose Alice wants to send an encrypted packet to Bob (who may be a security gateway); that packet must have an IP header with Bob's IP address as the destination (because, if it doesn't, the network won't forward the packet to Bob). Then, Bob decrypts the packet; if he (and Alice) were using transport mode, he retains the IP header. Hence, the decrypted packet will still have Bob's address as the destination. Then, if Bob is a security gateway, who does he forward the decrypted packet to? Well, the packet says that Bob is the destination, and so Bob it is (and even if Bob decides to ignore that rule, he doesn't have any idea who the real destination ought to be).
Now, actually, this isn't entirely true; there is a workaround. I have seen cases where Alice and Bob use a tunneling protocol between the two. In that case, you really can use IPsec transport mode. What Bob will do is, once he decrypts the packet, removes the tunneling header, which may expose the real plaintext IP header. Then, Bob knows who to forward the decrypted (and detunneled) packet to. This works; IMHO, this really is only a slight variation on tunnel mode, however apparently it does have some sort of advantage in some cases...
Update from RFC 4301. After careful re-reading of RFC 4301 I found proper clarification:
"In the case where transport mode is used between security gateways or between a security gateway and a host, transport mode may be used to support in-IP tunneling (e.g., IP-in-IP [Per96] or Generic Routing Encapsulation (GRE) tunneling [FaLiHaMeTr00] or dynamic routing [ToEgWa04]) over transport mode SAs. To clarify, the use of transport mode by an intermediate system (e.g., a security gateway) is permitted only when applied to packets whose source address (for outbound packets) or destination address (for inbound packets) is an address belonging to the intermediate system itself. The access control functions that are an important part of IPsec are significantly limited in this context, as they cannot be applied to the end-to-end headers of the packets that traverse a transport mode SA used in this fashion. Thus, this way of using transport mode should be evaluated carefully before being employed in a specific context. "