The answer is "no", in two ways.
First, the implementation of the algorithm could make use of side channels to leak data. The SSL timing attack permits an attacker who can execute multiple encryptions to "tease out" timing information that reveals bits of the key material. The original attack was based on the widely used OpenSSL implementation. Simply stated, if a particular key bit was 1, the encryption code would take one path through the code in a different amount of time than it would if the same bit was 0. By sending repeated requests for secure socket connections and timing the responses, the attackers were able to reconstruct the private key of the server.
Therefore, it would be possible for a malicious cryptographic implementation to deliberately include an asymmetric code path that would reveal the bits of the key via timing, not unlike Morse code. It would pass all test vectors perfectly, yet still be vulnerable. And not only would it be possible, but it would be easy. All a malfeasor would have to do is to implement an older version of the OpenSSL code, which has such a vulnerability already present. And he could mask that activity with feigned ignorance.
The other way a back door could get in is if the algorithm itself deliberately includes a back door, such as was suspected in the recent Dual EC DRBG random number generator scandal. Even a perfect implementation would yield a system that contains a back door.
There was long debate on whether or not the DES S-boxes contained such a back door, as the NSA had "strengthened" IBM's Lucifer algorithm back in the 1970s. It turned out that they actually did strengthen it against an attack (called differential cryptanalysis) that wasn't discovered until 1990 by civilian cryptographers Biham and Shamir. But an algorithm that incorporates deliberately weak S-boxes would also have a back door.