The (full) extended Euclidean algorithm is best expressed as a single loop with 6 variables in addition to inputs
- Input: polynomials $a$ and $b$ with $a\ne 0$.
- Setup: $(r,\hat r,s,\hat s,t,\hat t)\gets(a,b,1,0,0,1)$
- Invariant: $a\,s+b\,t=r$ and $a\,\hat s+b\,\hat t=\hat r$
- Loop: while $\hat r$ is not $0$
- $q\gets r/\hat r$
- $(r,\hat r)\gets(\hat r,r−q\,\hat r)$
- $(s,\hat s)\gets(\hat s,s−q\,\hat s)$
- $(t,\hat t)\gets(\hat t,t−q\,\hat t)$
- Output: $(r,s,t)$ such that $a\,s+b\,t=r$ and $r$ is $\gcd(a,b)$
Proof of correctness:
- The Setup phase is such that the invariant is initially verified.
- Variables $r$ and $\hat r$ evolve just as the two variables in the standard Euclidean algorithm. In particular, at each loop iteration, $\hat r$ becomes the remainder of the division of the former $r$ by the former $\hat r$; hence the degree of $\hat r$ strictly decreases at each loop iteration (if any). Hence the loop will terminate, with $r=\gcd(a,b)$ as in the Euclidean algorithm.
- Each loop iteration does $(r,s,t)\gets(\hat r,\hat s,\hat t)$; hence $\hat s+b\,\hat t=\hat r$ which holds before the loop becomes $a\,s+b\,t=r$ after.
- Each loop iteration does $(\hat r,\hat s,\hat t)\gets(r−q\,\hat r,s−q\,\hat s,t−q\,\hat t)$; hence after the loop the new value of $a\,\hat s+b\,\hat t-\hat r$ is the value that $a\,(s−q\,\hat s)+b\,(t−q\,\hat t)-(r−q\,\hat r)$ has before. We can rewrite this quantity as $(a\,s+b\,t-r)-q\,(a\,\hat s+b\,\hat t-\hat r)$, and using the loop invariant that's $0$. Hence $a\,\hat s+b\,\hat t=\hat r$ after the loop.
- Hence the invariant holds. Thus $a\,s+b\,t=r$ on output.
When we want the modular inverse of $b$ modulo $a$, we check that the output $r$ is a constant polynomial other than $0$, and the desired inverse is $t/r$, that is $t$ when working in $GF(2^k)$. In a computer implementation where we do not want to check the invariant, we can do without the variables $s$ and $\hat s$.
This technique is easy to program, because it uses a fixed number of variables. Contrast with the method in the question, where we have to keep all the result of the first phase computing the $\gcd$, and reuse them afterwards in a backtracking phase computing $r$ and $s$.
This technique is also easy when doing the computations by hand.
Example with $a=x^8+x^4+x^3+x+1$ and $b=x^7+x+1$.
$$\begin{array}{r|r|r|r}
r&s&t\\
\hat r&\hat s&\hat r&q\gets r/\hat r\\
\hline
x^8+x^4+x^3+x+1&1&0\\
x^7+x+1&0&1&x\\
x^4+x^3+x+1&1&x&x^3+x^2+1\\
x&x^3+x^2+1&x^4+x^3+x+1&x^3+x^2+x\\
1&x^6+x^2+x+1&x^7&x\\
0&\color{grey}{x^7+x+1}&\color{grey}{x^8+x^4+x^3+x+1}\\
\end{array}$$
This presentation avoids any duplication. We start by writing $a$ and $b$ in the top of the left column, and writing the constants $1,0$ and $0,1$ on their right.
On the right column, starting with the second line, $q$ is obtained by dividing the last two written terms on the left column.
New values are written on the first three columns by computing $r−q\,\hat r$, $s−q\,\hat s$, $t−q\,\hat t$ (where the variable with a $\hat\;$ is the most recently written one in the corresponding column, and the other is above).
We stop when a $0$ appears in he left column (and need not compute the two greyed terms on the right). The resulting $r$, $s$, $t$ are in the line above. When working with pen and paper, we can defer the computation of the second and third column until we have checked that this final $r$ is a constant polynomial, if that's desired.
If $a$ is irreducible and $b$ is not initially $0$, the final $r=\gcd(a,b)$ is always a constant polynomial, and always $1$ when working in $GF(2^k)$. This can be used to end the calculation and avoid the last line entirely.
When $b^{-1}\bmod a$ is thought, that is $t/r$, here $x^7$. The only use of the second column is checking that $a\,s+b\,t=r$ holds at each step.
An interesting variant of the algorithm does not compute $q$ exactly, instead keeping only it's high-order term. The number of steps tends to increase, but the computations are simpler.