A Modernized Enigma?

Question

I've seen answers here and elsewhere discussing how well the WWII Enigma could hold up to modern cryptanalysis, however, those answers always seem to assume the WWII conditions where the Germans had a limited number of rotor constructions, repeated patterns of plain text, and the Allies had access to a lot of the info about the rotors, their wirings and the limited combinations in use.

What if a software based enigma were executed with the following conditions:

1. 38 characters (versus 26)
2. ten rotors
3. the rotor settings, stecker settings, and rotor permutations themselves change with each message, all pre-planned using a cryptographically secure PRNG (i.e., the operator no longer chooses and transmits settings in a preamble)
4. assume (for sake of argument) no cribs and physical security - i.e., the attacker might have access to the encryptor, but not to any keys

How well would this hold up to modern cryptanalysis using super computers or networks? How long might it take to brute force it?

Answer 1

Looking at the possible setup combinations (key space): 10 rotors in random order each set to 1 of 38 positions is ~75-bits ( $\log_2(38^{10}\,10!)$ ). A plugboard of 38 is ~10-bits ( $\log_2\left(38 * 37\over2\right)$ ). The combined key would be ~85-bits of key material. Actually less because the rotors and plugboard suffer from a birthday attack.

AES supports key sizes of 128, 192 or 256 bits.

Thus such a machine would be substantially worse than AES just from brute force. Additionally there will probably be additional attacks which would make it even weaker.

Other issues:

1. If there is no crib how would you identify a correct decryption?
2. How will you securly share the "pre-planned" settings?
3. Settings were not transmitted in a preamble, they were in a pre-shared key sheet. A key sheet setting identifier could be sent in the message but that did not have the settings, it only identified the settings row in the current key sheet the receiver needed to already have.

Answer 2

I've seen answers here and elsewhere discussing how well the WWII Enigma could hold up to modern cryptanalysis, ...

Ehm, quote? I would say, the general opinion about the Enigma is actually the very opposite. The Enigma is the prime example why such a drastic change in the cryptographic mindset was necessary, where it is not enough if you can't break your own cryptosystem yourself.

Cryptanalysis has gone a long way since then, and today it is not enough to be resistant to ciphertext-only attacks. Back then, the attacks on the Enigma were made much easier by having some partial known plaintexts.
Today we even assume the attacker to have a lot of plaintext/ciphertext pairs. And known-plaintext attacks are the very least what we consider secure today.

---

So building a new scheme from an old-and-broken design is quite a bad idea. In today's binary world (in computers) using any other alphabet does't make sense.

And then there is one more thing: You suggest having an RNG responsible for the choice of the rotors. They are what provides confusion and diffusion in the Enigma... but randomly chosen functions are really bad at that. There is a reason, why those steps are never chosen randomly in block ciphers.

So to answer your questions: Probably you can break it with a known plaintext attack within seconds on a normal computer. Brute force or ciphertext-only attacks don't matter at all (unless brute-force is the best known stragety). So even if the brute-force attack cannot be done within reasonable time, it doesn't matter.