Can you tell me the 'theta' step of compression box
Keccak/SHA3 has no compression function, the inner function is a permutation.
In this step all the indices should be taken modulo 5.
Yes that is correct.
Do we have have to rotate it after taking mod 5 ??
And C[X-1] is left rotate or just putting the values of x as 0,1,2,3,4 and then taking mod 5.
C is a 64-bit temporary value, as is D. The index is done mod 5, not the value. X-1 mod 5 is the same as X+4 mod 5 for the index, addition is less likely to cause problems in code, and is generally easier to read, but is listed as subtraction in the specification for reasons which I hope are obvious.
And ROTC[X+1,1] is right circular rotation by 1 bit after taking mod 5, right??
Actually it is a left rotation on the 64-bit value by 1 bit. This is because the first bit is considered the least significant, so moving it by 1 bit to the next significant location is a left rotation when the least significant bit is on the right end of the value in memory.
If we look at theta without any loops, it looks like this:
C[0] = A[0,0] xor A[0,1] xor A[0,2] xor A[0,3] xor A[0,4];
C[1] = A[1,0] xor A[1,1] xor A[1,2] xor A[1,3] xor A[1,4];
C[2] = A[2,0] xor A[2,1] xor A[2,2] xor A[2,3] xor A[2,4];
C[3] = A[3,0] xor A[3,1] xor A[3,2] xor A[3,3] xor A[3,4];
C[4] = A[4,0] xor A[4,1] xor A[4,2] xor A[4,3] xor A[4,4];
D[0] = C[4] xor (C[1] <<< 1);
D[1] = C[0] xor (C[2] <<< 1);
D[2] = C[1] xor (C[3] <<< 1);
D[3] = C[2] xor (C[4] <<< 1);
D[4] = C[3] xor (C[0] <<< 1);
A[0,0] = A[0,0] xor D[0]; A[0,1] = A[0,1] xor D[0]; A[0,2] = A[0,2] xor D[0];
A[0,3] = A[0,3] xor D[0]; A[0,4] = A[0,4] xor D[0];
A[1,0] = A[1,0] xor D[1]; A[1,1] = A[1,1] xor D[1]; A[1,2] = A[1,2] xor D[1];
A[1,3] = A[1,3] xor D[1]; A[1,4] = A[1,4] xor D[1];
A[2,0] = A[2,0] xor D[2]; A[2,1] = A[2,1] xor D[2]; A[2,2] = A[2,2] xor D[2];
A[2,3] = A[2,3] xor D[2]; A[2,4] = A[2,4] xor D[2];
A[3,0] = A[3,0] xor D[3]; A[3,1] = A[3,1] xor D[3]; A[3,2] = A[3,2] xor D[3];
A[3,3] = A[3,3] xor D[3]; A[3,4] = A[3,4] xor D[3];
A[4,0] = A[4,0] xor D[4]; A[4,1] = A[4,1] xor D[4]; A[4,2] = A[4,2] xor D[4];
A[4,3] = A[4,3] xor D[4]; A[4,4] = A[4,4] xor D[4];
Where A[x,y] is one of the lanes of the state with row index 'x' and column index 'y', and '<<<' is a left rotation of the value to the left by the bits to the right.