How to adapt the equation of Gaussian mechanism noise based on number of executions

Question

I'm trying to build a differentially private machine learning model. I'm using the Gaussian mechanism to calculate the required noise amount based on pre-defined privacy budget value 𝜖

The equation to find the noise amount is below

$\sigma = \sqrt{2ln(1.25/𝛿)} / \varepsilon $

I'm aware that when the model performs several epochs, say 15, I will need to use either the naive composition or the advanced composition theorem to calculate the new $\varepsilon'$

$$\varepsilon' = \sqrt{2k\log(1/\delta)}\varepsilon + k \varepsilon (e^\varepsilon-1)$$

1. I'm interested to know how to adapt the first equation and use it to compute the total amount of noise to be dded based on the number of executions (epochs)

Answer 1

The formula you mention to get $\sigma$ given $\delta$ and $\varepsilon$ is only correct for $\varepsilon<1$. It's also not tight.

If you use Gaussian noise on multiple statistics, each of sensitivity 1, and want to get the tightest possible privacy guarantees, then you should use the Analytic Gaussian Mechanism instead (Theorem 8), replacing $\Delta$ with the square root of the number of your statistics. It's tight and works for any value of $\varepsilon$ and $\delta$.

The only problem with that approach is that there's no formula giving $\sigma$ as a simple function of $\varepsilon$, $\delta$ and $\Delta$; so you have to do some binary search to find the best possible value (and be careful about how you do the floating-point calculations). Here is an example from our open-source differential privacy library.

Finally, if what you want to do is train a model in a DP way, I would suggest using existing tooling, like TensorFlow Privacy, rather than rolling your own.