Update: I made my lattice attack worked finally. As the actual reason is quite complicated I decide to write an answer below to describe how it worked so anyone with similar question might get inspiration from my work. The Question is not modified.
I was studying lattice attack recently. I tried to use data from TPM-FAIL to help me understand this attack and try to implement an attack using "textbook method" (there are some optimization in TPM-FAIL attack scripts). I had some questions reading the materials that I could not figured out myself. Please help me if you have any idea.
The DSA signature formula can finally be transformed to following equation:
$k_i-s_i^{-1}r_id-s_i^{-1}H(m)\equiv 0 \pmod{n}$
where k is the ephemeral key, (r,s,) is the signature pair, d is private key, H(m) is message digest as usual... you know the drill.
Then it can be transformed in to lattice form. Something like following equation:$k_i+A_id+B_i\equiv 0\pmod n$, where $A_i=-s_i^{-1}r_i$ and $B_i=-s_i^{-1}H(m)$
What I don't understand is that, why does it have to be negative? actually tried to modify the attack script provided in TPM-FAIL dataset and found that removing -1 in A_i and B_i will make the attack failed. But we can rewrite the first equation as:
$s_i^{-1}r_id+s_i^{-1}H(m)\equiv k_i\pmod{n}$
The concept of lattice attack should still hold: If the lattice vector is small, basis reduction algorithm should generate the answer. What have I got wrong?
The second thing is that I tried using the un-optimized approach, the SVP lattice is built like below:
$\begin{bmatrix}n&&&&&\\&n&&&&\\&&\ddots&&&\\&&&n&&\\A_1&A_2&\dots&A_t&K/n&\\ B_1&B_2&\dots&B_t&&K\\\end{bmatrix}$
But we can clearly see that K/n will be a fractional number which cannot use BKZ() or LLL() method in sagemath. What I understand is that we can multiply every thing in this matrix by n so everything in this matrix is integer. After apply BKZ() we divide everything by n to recover the original result: the private key should be in the second last column of the first row. However I failed to recover the private key. Even if I used 100 signatures with 8-bit leakages. Is my approach correct?
Thank you for your help in advance!