A modular multiplication algorithm invented by Peter Montgomery that allows modular arithmetic to be performed efficiently when the modulus is large (typically several hundred bits).

The Montgomery reduction algorithm Redc(T) calculates TR^{-1} bmod {N} as follows:

m := (k (T mod R)) mod {R}
t := (T + mN)/R
if  t >= N return t - N else return t.

Using this method to calculate c is generally less efficient than a naive multiplication and reduction, as the cost of conversions to and from residue representation (multiplications by R and R^{-1} modulo N) outweigh the savings from the reduction step.

The advantage of this method becomes apparent when dealing with a sequence of multiplications, as required for modular exponentiation (e.g. exponentiation by squaring).

Many important cryptosystems such as RSA and DSA are based on arithmetic operations, such as multiplications, modulo a large number. The classical method of calculating a modular product involves first multiplying the numbers as if they were integers and then taking the modulus of the result. However, modular reduction is very expensive computationally—equivalent to dividing two numbers.

When several modular successive multiplications are required modulo the same modulus, as in modular exponentiation, the saving in modular reduction thanks to Montgomery reduction often outweigh the cost of conversion to and from Montgomery form.