A fast hash is by definition easier to attack. A modified dictionary attack (a dictionary with symbols inserted etc.) would easily filter out the passwords that would generate the adler32 and then the second, slow hash can be used to validate if the correct password was used.
It would be hard(er) to use a rainbow table attack though as rainbow table attacks won't work on hashes that have a lot of collisions, and the actual password hash is protected by a salt. A rainbow table would not be very efficient for a small user base anyway.
If you'd use a smaller hash output then there the chance of a collision indeed becomes higher. That would mean that you'd have to reject specific passwords as they would hash to the same account. Normally in a hash table you'd simply look for the correct key that generated the hash. But in your case that would be the password itself, and the whole idea of a password hash is to not store the password itself. There are some ugly workarounds possible of course, but let's not over-complicate things. This idea doesn't really remove the major drawback of being able to filter out possible password matches.
You could indeed use a randomly generated "secret" rather than a password. In that case you might just store SHA-256 hashes over that secret, as nobody is going to guess a 40 character secret anyway. Nobody is going to remember it either of course; you'd be restricted to using a password manager basically.
There may be an Oracle attached to your solution. Basically the hash acts like a user ID. However, if the system returns a different error or takes less time returning an "unknown password" if the adler hash cannot be found then an online adversary can use your scheme to quickly filter out incorrect passwords as well.
In general: there is a good reason why this kind of scheme is not common. There are simply too many issues. Don't forget that the password would still identify the user in your scheme, so your user doesn't gain any anonymity. If you want that there are several anonymous / identity-free PAKE- and ZK-schemes out there. Those do require a higher proficiency in cryptography though.