If I have some cryptosystem that is homomorphic and supports addition,
from my understanding, I can do this:
E(A) + E(B) = E(A+B)
Yes, anyone that hold the ciphertexts E(A) and E(B) can end up with an encryption of A+B.
But please, make sure you understood that the addition on E(A) and E(B) (ciphertext domain) may not be the same that the addition you perform on A and B (plaintext domain). Also, this is not really an equality, since the scheme may be not deterministic, so, the right way to write this is "Dec(E(A) + E(B)) = A + B".
And then I can decrypt E(A+B) without knowing either A or B.
This is the point you are misunderstanding... Only the person possessing the decryption key can decrypt.
You say "I can calculate", "I can decrypt", etc... Maybe, to avoid confusion, you should fix some parties. What is is typically done when homomorphic encryption schemes are presented is to define
- Data owner: is the party that has the data to be encrypted and (generally) generates the keys $pk$ (encryption key) and $sk$ (decryption key).
- Third-party: is anyone that receives the encrypted data, the key $pk$, and whose role is to store and compute over the encrypted data. This third-party is, in general, presented as some cloud service.
Thus, the data owner can encrypt, decrypt, and compute homomorphically while the third-party (or, anyone else), can only compute homomorphically, but can't learn the results of those computations.
As fgrieu said in her/his answer, homomorphic schemes may be symmetric. In this case, $pk = sk$ and, of course, the third-party can't receive $pk$. Therefore, the cloud can't encrypt anymore, which means that all the auxiliary values used in the computation have to be encrypted by the client and send to the cloud as well. For instance, to homomorphically compute the average of $n$ numbers, the client would have to encrypt also the auxiliary value $n$ and send $E(n)$ to the cloud...
Furthermore, deterministic homomorphic encryption can't even be IND-CPA secure. So, this is way randomized homomorphic encryption is studied.
And the last point: the scenario I have described is not the most general one, because it is common to have a third key, called evaluation key, which is used to perform homomorphic operations (so, it has to be submitted with the data to the third-party).