How we can measure key space in transposition algorithms? Should we specify the method, like rail fence?
Measuring the size of the key space in transposition algorithms is not important, because their security is far less than the size of the key space would suggest. Therefore, any measure of effective key length will be misleading and will not give an accurate picture of the true security of the scheme.
In general the standard way to compute the effective key length of a cryptographic algorithm is as follows: we let $K$ denote the set of all possible keys for that algorithm; then the effective key length is $\lg |K|$, where $|K|$ denotes the size of the set $K$ and $\lg$ denotes the logarithm to base 2.