Yes, there are several ways in which Mallory could pretend to be Amy.
One obvious way, which doesn't even involve Amy herself in any way, would be for Mallory to perform steps 1 and 2 of the protocol normally, as if he were Amy. Then, given Betty's nonce $n_b$, Mallory can start a second, parallel instance of the protocol, again pretending to be Amy, and submit $n_b$ as Amy's nonce. The response from Betty then contains $E(k, n_b)$, which Mallory can send back to Betty to complete the original authentication protocol:
- M(A) → B: $n_a$ (= arbitrary nonce chosen by Mallory, pretending to be Amy)
- B → M(A): $n_b \;\|\; E(k, n_a)$
Second, parallel session:
1'. M(A) → B: $n_b$
2'. B → M(A): $n_b' \;\|\; E(k, n_b)$
- M(A) → B: $E(k, n_b)$
Of course, if Amy is also willing to respond to authentication attempts initiated by Betty (or by Mallory pretending to be her), then Mallory can perform the parallel steps 1' and 2' with Amy instead of Betty. Thus, even if Betty's smart enough to check whether the nonce she receives in step 1' matches her own earlier nonce (or even if she simply refuses to participate in more than one authentication exchange at a time), she will not be able to detect this version of the attack.
As owlstead suggests, the way to defeat this attack is to disambiguate the encrypted values sent in step 2 and 3, so that the value from step 2 cannot be used by Mallory in step 3. Some possible ways to do this would be to:
Include a step number in the encrypted strings, e.g.:
- A → B: $n_a$
- B → A: $n_b \;\|\; E(k, 2 \;\|\; n_a)$
- A → B: $E(k, 3 \;\|\; n_b)$
Include both nonces in each encrypted strings (in different order), e.g.:
- A → B: $n_a$
- B → A: $n_b \;\|\; E(k, n_a \;\|\; n_b)$
- A → B: $E(k, n_b \;\|\; n_a)$
Include the (assumed) identities of the sending and receiving parties in the encrypted strings (NOTE: does not protect against the modified attack where Mallory contacts Amy instead of Betty for steps 1' and 2'), e.g.:
- A → B: $n_a$
- B → A: $n_b \;\|\; E(k, id_b \;\|\; id_a \;\|\; n_a)$
- A → B: $E(k, id_a \;\|\; id_b \;\|\; n_b)$
All of the above, e.g.:
- A → B: $n_a$
- B → A: $n_b \;\|\; E(k, 2 \;\|\; id_a \;\|\; n_a \;\|\; id_b \;\|\; n_b)$
- A → B: $E(k, 3 \;\|\; id_b \;\|\; n_b \;\|\; id_a \;\|\; n_a)$
Personally, I would recommend the last option, if only because it makes the encrypted strings self-documenting: $E(k, 3 \;\|\; id_b \;\|\; n_b \;\|\; id_a \;\|\; n_a)$ can be read as "Amy ($id_a$) authenticating with nonce $n_a$ to Betty ($id_b$) with nonce $n_b$ in step 3", which is exactly what the message is supposed to communicate.
Also, obviously, the encryption method $E$ needs to be non-malleable, and the combining operation $\|$ should not introduce any ambiguity. In fact, calling $E$ "encryption" is somewhat misleading here: the protocol works fine even if $E(k,m)$ is just a MAC, but has obvious attacks if $E$ is a malleable (but still semantically secure) encryption scheme like CTR mode encryption, as Mallory can simply take $E(k, n_a)$ and turn it into $E(k, n_b)$ by flipping the appropriate bits.
The protocol as described also has other potential issues, depending on how it's used. For example, if Mallory can insert himself as a middle-man between Amy and Betty, then he can simply relay any messages he receives from Amy to Betty, and vice versa, thereby trivially convincing both that they're talking directly to each other, even though both are actually talking to Mallory.
Fundamentally, without some extra out-of-band information, there's no way to prevent this attack from within the authentication stage of the protocol: as long as Amy and Betty just receive messages from some unspecified network, there's no way for them to tell whose hands they've passed through.
The only solution is for Amy and Betty to encrypt their subsequent communications using the shared key $k$ (or, better yet, an ephemeral session key derived from $k$ and the nonces $n_a$ and $n_b$, and preferably also the identities $id_a$ and $id_b$) with a secure, non-malleable encryption scheme to ensure that, even if Mallory can intercept their messages, he will not be able to decrypt or alter them. (Of course, the messages should also carry sequence numbers / timestamps and sender / receiver IDs to prevent replay attacks.)