The SRP paper has this point in its list of security properties:
6. If the user's password itself is compromised, it should not allow the intruder to determine the session key K for past sessions and decrypt them. Even present sessions should at least be protected from passive eavesdropping.
The following section is titled Reduction to Diffie-Hellman and shows that, given an algorithm which retrieves the session key from the transmitted values and the password, you can also solve the computational Diffie-Hellman problem.
So yes, SRP is as secure as Diffie-Hellman for key exchange if the password is known to the attacker (and the other parameters are chosen properly, i.e. uniform random in their respective domains).
On the other hand, SRP with a non-secret password has the same problem as the pure Diffie-Hellman key exchange, when facing an active attack, i.e. a man-in-the-middle or an attacker who impersonates only one of the two parties.
I would like to do key exchange under the assumption that I've already
done mutual authentication (through some out of band user intervention).
Sorry, this rings alarm bells. Depending on how this out of band user intervention looks like, it could prove nothing at all about your actual communication going on, or about the non-existence of a man-in-the-middle attack.
Your "out of band authentication" should at least verify some value derived from the negotiated secret $K$, or (if you want it to go on beforehand) negotiate some (real) secret which then goes into the process as a part of the "password", so the attacker actually doesn't know the password, and can't mount a man-in-the-middle attack.