If you're looking to test the parameters for an RSA or ElGamal implementation, well, that's fairly straightforward (assuming you can look at the innards of the implementation, and ask "are these primes drawn from a random distribution, do you get the RSA padding right, are you secure against the relevant side channel attacks, etc). You really do need to know your crypto, but it's not that hard.
On the other hand, if you've devised your own public key cryptosystem, well, that's much harder. Actually, symmetric cryptosystems aren't that much easier; the 'avalanche' test that you alluded to is a very weak test; a cipher can easily pass that with flying colors, and still be very weak.
In both cases, the real test is 'can a clever individual who has access to the ciphertext, possibly some plaintext, and (in the case of public key cryptosystems) the public key find an attack'. That's not an easy question to answer (or so you hope, if there is an easy answer, it's be 'yes, here's this attack').
With public key cryptosytems, the obvious place to start is the 'hard problem' they're based on; why precisely do we think it's hard. For RSA, well, it's the "RSA problem" (typically, people say factoring, however that's not proven to be true); for ElGamal, it's the Diffie-Hellman problem. In your case, you need to consider what problem makes it easy (with a public key) to perform in one direction, and hard (with a public key, but not the private) to perform in the other direction.
Issues with the hard problem aren't the only possible way a system can fail (consider RSA with no padding; the RSA problem remains hard, but we can find ways to break the system by taking advantage of the homomorphic properties of raw RSA); however it's the obvious place to start.