I'm researching cryptography for a school project, came across the above two ciphers, and something occurred to me. Would combining these two schemes give a considerably stronger encryption than either one individually?
Consider two encryption schemes: one in which only the vigenere cipher is used (which I'll call $E_1$), and one in which both the vigenere and playfair cipher is used ($E_2$).
When discussing $E_2$, assume all references to it's key will be referring to the key of it's vigenere cipher unless explicitly stated otherwise.
When trying to decrypt $E_1$, the first step is looking for repeated sequences of characters and recording their distance apart to help determine the length of the key, which is likely to be a factor of many of these recorded distances.
However, with an $E_2$ cipher on the same ciphertext, this won't work nearly as well. If E1 showed repetitions of odd length n, $E_2$ would show those same reptitions of length $n-1$ with either the final character or the first character missing, while even length repetitions would either be reproduced at length n or have both the end and the beginning cut off for length $n-2$. In addition, any given sequence will become one of two entirely different sequences based on where the beginning of the sequence falls at the start of a playfair pair or not.
This means two things:
Repeated sequences aren't quite as clear - particularly shorter ones (which are the most common): sequences of length 3 (notably the word 'the') for example, become one of two sequences of length two, which is considerably easier to form by chance (which is why repeated sequences of length 2 are often not used in vigenere cryptanalysis). Four letter repetitions suffer a similar fate - half the time, the playfair pass will instead make them into repeated sequences of length two as well.
If $E_1(P)$ produces n repeated sequences (a single repetition) of average distance m apart, $E_2(P)$ will, on average, produce 2 different repeated sequences, each with $n/2$ repetitions that are $2*n*$ distance apart. This means that the distances between the repetitions are significantly larger and could have significantly more factors, which would mean the length of the key is less certain to the cryptanalyst.