If \(A=B\), the answer is \(1\). Below, assume \(A \neq B\).

If \(\mathrm{gcd}(a,b) \neq 1\), then \(a\), \(b\), and \(g\) during the subsequent operations will always be multiples of \(\mathrm{gcd}(a,b)\), so we may replace \(a\) and \(b\) with \(\frac{a}{\mathrm{gcd}(a,b)}\) and \(\frac{b}{\mathrm{gcd}(a,b)}\), respectively, and subtract \(1\) from \(a\) and \(b\) instead of the original operation. (This is not mandatory, but it slightly simplifies the handling of remainders.)
If we find the smallest \(t\) such that \(\mathrm{gcd}(a-t, b-t) \neq 1\), we can perform multiple operations together to speed up the solution. Here, the operation does not change \(|a-b|\), so \(\mathrm{gcd}(a-t, b-t)\) can only be a multiple of \(|a-b|\). Therefore, \(t\) is the smallest value of \(a \bmod x\) for a divisor \(x\) of \(|a-b|\) that is at least \(2\).

One can divide \(a\) and \(b\) by values that are at least \(2\) only \(\mathrm{O}(\log a)\) times, so the above solution has a complexity of \(\mathrm{O}(\sqrt{N} \log N)\), where \(N = \max(a, b)\). Additionally, it is enough to consider divisors of \(|A-B|\) instead of those of \(|a-b|\), so it can be improved to \(\mathrm{O}(\sqrt{N} + d\log N)\), where \(d\) is the number of divisors of \(|A-B|\). Furthermore, we only need to consider prime factors, so it can also be \(\mathrm{O}(\sqrt{N} + p\log N)\), where \(p\) is the number of prime factors of \(|A-B|\).