- You're okay with the encrypted information being easily decrypted after the session
- You have a guarantee that the session won't be very long
- You use random and unpredictable safe primes $p$ every session
I think it should, theoretically, be okay to use slightly smaller bit lengths for $p$.
That said, in practice it is probably a bad idea, and it's better to use ECC as @CurveEnthusiast mentioned. Generating safe DH primes is non-trivial, judging by how many implementations screw it up. And in the case of smaller groups, it is absolutely crucial that you generate random and unpredictable safe primes for every session (otherwise, someone can simply do a large amount of precomputation on the group, and crack connections using that group very, very quickly -- see Logjam).
What is your threat model? How much computing power does your attacker have at his disposal?
For reference, the authors of the Logjam attack in 2015 cracked several 512 bit groups in a week. They estimated that this could be optimized by a factor of 3, and they deliberately spent more time on precomputation. How fast do you think a state actor could break a 512 bit key exchange? It's not inconceivable to think that with a huge amount of computation power, and some very clever optimizations, a state-level adversary could do it in a few hours.
In short, it's not worth the risk. You run a big risk of improperly generating the weaker DH groups. You run another big risk of reusing the weaker DH groups, or having them be predictable, thereby opening yourself up to precomputation attacks.
Try ECC curves instead, and see if they don't alleviate your time/power consumption issues.