In my opinion, the question's
1) Perform deep learning on homomorphic encrypted data
should not be considered without feedback from the holder of the decryption key. I have the intuition that it is impossible, and to my knowledge it was never even attempted. Learning is typically performed on clear data, and "learning on (..) encrypted data" seems to require at least some use of the decryption key.
That's because ciphertext yields no exploitable information for someone/something neither holding the key nor breaking the cipher. Without some access to the decryption key, any data processing on the ciphertext is nonsensical (except for cryptanalysis), even when the encryption is homomorphic. Homomorphic encryption will let one compute some function from the encrypted data, but the result stays encrypted, and no conclusion can be reached without the decryption key. I thus do not see how, without the key, decisions characteristics of deep learning activity (to connect a neuron or not, or other long-term change) could be taken meaningfully (better than randomly); or how positive learning feedback (where earlier learning enables new progress) could occur.
The best I see possible without the decryption key would be to compute encrypted values from encrypted training data, then use these encrypted values as parameters of a predefined model. That way, the encrypted data used for training would meaningfully influence the model. But is that learning?
Rebutal for comment, linking to Ehsan Hesamifard, Hassan Takabi, Mehdi Ghasemi's CryptoDL: Deep Neural Networks over Encrypted Data as argument that 1) can be done. Quoting that paper:
We assume that the training phase is done on the plaintext data and a model has already been built and trained
This (and other) literature builds deep learning networks that can process data encrypted with homomorphic encryption without holding the key; but I fail to see how something could meaningfully learn from well-encrypted data without using the decryption key, which is how I read the question's 1).
There is no such problem with 2), where deep learning was performed on clear data, and the model encoded for homomorphic encryption. Partially homomorphic encryption will not be very useful, for it won't allow the non-linearity present in any useful model. Fully Homomorphic Encryption seems a must.