- Is this actually a viable method of key exchange?
No. An eavesdropper can find the integer $b$ chosen by Bob from $x$ (as sent by Alice) and $b'$ (as sent by Bob), and the equation $b'\,=\,b\,x\bmod 1$ (meaning $\exists d\in\mathbb Z,\ b'+d=b\,x$). The shared $k$ can then be determined from the $a'$ sent by Alice, just as Bob does.
If we take the numbers in the paper's example (section 5), given the first 14 decimals $B$ of $b'$ (out of the 61 sent by Bob) and the 24 first decimals $X$ of $x$ (out of about 71 received by Bob and 30 shown), we get $b$ as follows:
- $B$ is the 14 first decimals of $b'$, thus $\big|B-10^{14}\,b'\big|<1$
- $X$ is the 24 first decimals of $x$, thus $\big|X-10^{24}\,x\big|<1$
- $b'+d=b\,x$ implies $10^{24}\,(b'+d)=10^{24}\,b\,x$
- thus $10^{10}\,(10^{14}\,b')+10^{24}\,d=(10^{24}\,x)\,b$
- thus $\big|10^{10}\,B+10^{24}\,d-X\,b\big|\le10^{10}$
One method to solve such Diophantine inequality for $b$ (we don't care for $d$) uses continued fractions¹. But we can be lazy and ask Wolfram in near-natural language
|10^10*62932060503117+10^24*d-693147180559945309417232*b|≤10^10 and b>0 Diophantine
and it gives us $b=6913952452$; or use Mathematica to the same effect, Try It Online!
B = 62932060503117; (* 14 first decimals of b' *)
X = 693147180559945309417232; (* 24 first decimals of x *)
Solve[Abs[10^10*B+10^24*d-X*b]≤10^10 && b>0 && b<10^11, {b,d}, Integers]
…
{{b -> 6913952452, d -> 4792386648}}
More generally, if $a$ and $b$ have $n$ digits, and we want the shared $k$ to agree over $m$ digits with residual odds of the contrary in the order of $10^{-j}/5$ (where that $/5$ assumes rounding to the nearest), then Alice and Bob should agree on $x$ over $2n+m+j$ digits, and send $a'$ and $b'$ over $n+m+j$ digits.
The paper's example targets $m=30$ shared digits in $k$, uses $j=21$ (thus almost always reaches its target), and $n=10$, which is small enough that brute force is usable to find $b$. But the attack above (or the explicit method¹) works in time only polynomial in the number of digits used. Try It Online! with $m=50$, $j=21$, $n=70$: it still takes under 1 second to recover $b$.
The paper should have been screened by the review process of a would-be scientific peer-reviewed journal on cryptography. I conclude Article Processing Charges do not ensure quality. I am not aware of similar mishaps in peer-reviewed publications of the IACR (thus excluding eprint), but that happens more than occasionally in many other publications. See Wikipedia's article on the publisher, and the publisher's response.
- Is it really resilient to quantum computers?
In a sense yes for today's quantum computers, since they are unable to compute exact results on more than a few bits. In a distant future, perhaps no. That is: the scheme is not quantum-resilient. It's not even resistant to classical computer.
- Is the $\bmod1$ calculation a known one-way/symmetrical function, and is it used elsewhere?
In cryptography, schemes that manipulate reals to a precision better than an FPU register are the exception rather than the norm, and always are internally implemented as integers divided by some integer scaling factor (often a power of a base). Therefore $\bmod1$ is implemented as modular reduction modulo the scaling factor.
Modular reduction modulo an integer is a common building block for one-way functions, such as (in Diffie-Hellman) $k\mapsto g^k\bmod p$ where $p$ is prime, $q=(p-1)/2$ is prime, and $g^q+1\bmod p=0$. But modular multiplication by a public constant (as used in the question's article) is not one-way.
Following the author's comment: the question, and this answer, assume $x$ is public. If $x$ is secret, the scheme is indeed less insecure, and even secure on first use of $x$ when the right parameterization is used. But with the parameterization in the example, we don't need $x$ to find the shared value². And even with good parameterization, if $x$ is reused, security quickly breaks down. The second part of the article's “The security benefits from distributing $x$ via a secure channel, but this is not necessary” is untenable.
¹ It took me more time than I had hoped to reconstruct an explicit method:
- Using continued fractions, write $\frac X{10^{24}}$ as the simplest fraction $\frac r s$ such that $\left\lfloor\frac{10^{24}\,r}s\right\rceil=X$. This is essentially the most compact representation of $x$ as an integer fraction giving it to 24 decimal places. Here $\frac r s=\frac{698526063389}{1007760087583}$
- Round $\frac{s\,B}{10^{14}}$ (here 634204188043.9984…) to the nearest integer $t=\left\lfloor\frac{s\,B}{10^{14}}\right\rceil$.
- Without proof: there remains to solve for $b$ the Diophantine equation $s\,d+t=r\,b$, here $1007760087583\,d+634204188044=698526063389\,b$
- Reducing modulo $s$, we get $t\equiv r\,b\bmod s$, with $r$ and $s$ coprime by construction of $\frac r s$, thus $r^{-1}\,t\bmod s=b$ when $b<s$ (without proof: which holds with high probability when enough digits of $x$ are known in order for the scheme to work reliably). Here $b=6913952452$.
² Assume we have $a'$ and $b'$ to 61 digits as in the example, but not $x$. :
- Using continued fraction, express what's known of $a'$ (61 digits each) as a reduced fraction $a'=\frac e f$ with $f^2<10^{61}$, same for $b'=\frac g h$.
- From $a'+c=a\,x$ and $b'+d=b\,x$ with $0<x<1$, for at-most-10-digit $a$ and $b$, it holds the Diophantine inequality $\big|b\,h\,(e+c\,f)-a\,f\,(g+d\,h)\big|<2\cdot10^{10}$ with $0\le c\le a<10^{10}$, $0\le d\le b<10^{10}$
- Reducing this $\bmod f$ and $\bmod h$, it comes that for some unknown integer $r$ with $\big|r\big|<2\cdot10^{10}$, it holds $a\,g\,f\equiv r\pmod h$ and $b\,e\,h\equiv-r\pmod f$.
- With $\ell\gets\gcd(f,h)$, $f'\gets f/\ell$, $h'\gets h/\ell$, and $s$ defined as $r/\ell$, it comes that for some integer $s$ with $\big|r'\big|<2\cdot10^{10}/\ell$, it holds $a=((g\,f')^{-1}\,r'\bmod h')$ and $b=((-e\,h')^{-1}\,r'\bmod f')$.
- With $g'\gets(g\,f')^{-1}\bmod h'$ and $e'\gets(-e\,h')^{-1}\bmod f'$, it comes $a=(g'\,r'\bmod h')$ and $b=(e'\,r'\bmod f')$.
- This now can be solved by Mathematica, Try It Online!. This gives 4 possible values for the shared $k$, and 10 for $x$.