The tricky part is what if the agents do not follow the protocol? It is often the case that by deviating from the protocol, an agent can gain a higher payoff than following the protocol. if this happens, then the agent may not follow the protocol. One party deviating from the protocol may cause other parties to change their strategy. Will they reach a stable state?
Therefore, to prove security, you would need to specify the following:
- All the parties (players);
- All actions can be performed by the parties (strategies);
- Is there any sequence or order of parties have to act, or they move simultaneously;
- What information do the parties have in the process? Do they know everything or only some information?
- The utility functions of each party -- taking into account not just money, but others types of payoffs.
- All assumptions
Putting those together, you can formally specify a game. Having the game formally specified, you will have all possible outcomes induced by all possible combination of parties' strategies, and payoffs for each party associated with the outcome. You can then analyse the game to see whether there exist an equilibrium (or more than one), which is a "solution" of the game. An equilibrium consists of a set of strategies for each party (and maybe probabilities). An equilibrium basically says if all parties choose the strategies in the equilibrium, they will not want to change their strategies because their utilities won't be better.
In the simplest case, if you have one equilibrium and it entails your security goal, then you can say you have proved security in the presence of rational parties. But the situation can be much more complicated.