Well, whether there's a better way depends on what you mean by 'better'; that is, what are you actual requirements? On the other hand, depending on what you are doing with the EC key, it may be that the overall design your assuming (map a passphrase to an EC private key) is itself unsuitable (and the actual function you use doesn't actually change that).
If you're asking about whether to use a key derivation function, the obvious question is 'as opposed to what?'. To convert a passphrase to an EC private key, what you're doing is mapping a text string into an integer between 1 and $q-1$ ($q$ being the order of the curve), if you don't use a key derivation function, you're going to use something that looks a lot like a key derivation function (even if we might use different terminology to describe it; such as 'seeding a CSRNG with tha passphrase as a seed, and using that output').
Instead, what you should be asking is 'what do I need from a key derivation function right here'.
Now, that, I can't immediately answer for you; you need to consider what you need. For example:
Do you care how fast the EC private key is generated? How fast does it need to be compared to the performance of the device you're generating it on?
How much do you care about dictionary attacks against the EC private key? If someone gets the public key, and then runs through a dictionary to try to obtain the private key, how difficult should that be?
Do you care about precomputational attacks; that is, if the attacker is allowed a large amount of computation before hand, is he able to build a table to make recovering the password (and private key) from a public key much faster?
Depending on your answers, a key derivation function (or similar) might not be at all suitable. For example, if you are using these EC keys are credentials (so that they need to be immune to any dictionary-style attacks for extremely long periods of time), then this design (which maps a passphrase to a EC private key) might not be suitable at all, because it does nothing to prohibit an attacker from scanning through plausible passphrases; you may need to rethink your approach. For example, you might require some long term secret (that you stir into the key derivation process); this long term secret would prevent any possibility of a dictionary attack (because the attacker wouldn't have that).