Yes, but you probably need to use a cryptographically secure RNG from your OS. Otherwise the input to the signature generation function may be mathematically linked, and the security of the system is not guaranteed. It is of course wise to use a 256 or higher curve, otherwise the security provided by the random number may be lowered.
Besides compressing the input, the hash is used to generate pseudo-random input to the signature generation function. It removes any mathematical links between the input values as cryptographic hash functions are one-way. However, using a small enough, random / independent values should already meet these requirements.
Random number generation by the OS is likely to be significantly slower than performing a single hash over a single block of input.
In that sense you could think of using 128 bit numbers instead of 256 bits and perform full ECDSA on that. You could also think of using deterministic ECDSA as specified in RFC 6979. These changes may well have a larger impact than removing the relatively fast hash from ECDSA. And you'd still be standard-compliant to boot; all of the security requirements on ECDSA would still be met.