I have a program that uses a custom algorithm to encrypt a message
This algorithm is called a cipher. There are plenty well known ciphers that are considered secure. If you have a "custom cipher" then you're either using something of your own design. Using proprietary schemes is not recommended except for learning purposes.
My program support multiple passes. How many iterations of these passes my program should do? Is there any ideal number in the crypto world? I have read a little about three pass protocol. Does that apply in this case?
Any modern cipher should be secure with one pass.
Most block ciphers already contain a high amount of rounds internally. If you use the same key you're basically just multiplying that amount. If a break is found for all rounds of the cipher then it probably breaks multiple passes as well.
If you want to consider choosing a number of rounds for your cipher then you need a key schedule and a way to connect the inner functions that use the sub-keys. Cipher design is a large topic of which entire books are written (such as "The Design of Rijndael", the cipher that became AES).
If the key needs to be used multiple times - in different ways - then the key can be split into sub-keys using a *key-schedule. The key schedule is internal to a cipher.
The three pass protocol doesn't seem to have anything to do with message encryption directly.
Any issues or trade-offs can you think of with this type of approach?
Once you have a secure cipher then you can focus on the protocol and system rather than the cipher. There are plenty secure ciphers - such as AES / GCM for symmetric encryption or RSA in OAEP mode for public key encryption. The trick is to apply them well.
Note that stream ciphers or block ciphers in streaming mode may produce a key stream and XOR that with the plaintext. If you would apply those ciphers multiple times using the same key and IV then you would either completely negate encryption an even number of encryption attempts or only apply it once for an odd number of encryption attempts.
Check out how a one-time-pad works and try to apply the encryption twice to understand what I mean.
This also shows that we cannot advise how to construct multiple rounds within your cipher without understanding the design. For all we know, we could be destroying security rather than improving it.
