As pointed by the question, in SRP an attacker knowing the verifier can impersonate the server. That's not against the security objectives of SRP.
To carry the attack, the attacker also needs to know the salt, but it is public and can be obtained from the server. With both verifier and salt, the attacker then behaves with respect to the client just as the server does.
The problem of authenticating the server can be solved by https and a server certificate, inasmuch as you trust certificates (certificates only checked by a common web browsers/OS should be only marginally trusted: governmental and other well-founded adversaries can easily obtain a forged certificate with any CN field they wish, and certificate chain, that pass this test). It is still reasonable to use SRP on top of https with a server certificate: at least, if the certificate is subverted, the password does not leak, and the user can't be impersonated to the server (the main goal of SRP), contrary to what would happen with the password sent over https.