Cryptography Stack Exchange is a question and answer site for software developers, mathematicians and others interested in cryptography. It only takes a minute to sign up.
Sign up to join this community
Is there a way to 'mangle' a public data-source (for example, the current date in YYYYMMDD or the top New York Times headline) to form a one-time pad that will sufficiently hide the pad's source? What other issues would such a scheme be running into?
Disclaimer: I am not an expert cryptographer and am just interested in the matter - I do not intend to implement this cryptosystem.
Very short answer: No
Quite Short answer: No, because a scheme can only be a One-Time-Pad if the entire pad is perfectly random and secret.
Concise answer: It sounds like you're trying to build a stream cipher. The security of it really comes down to how much of the scheme you think can be kept secret. If I listen in to your wifi and hear you requesting a certain resource, then I can work out what your data-source was, calculate the pad myself and decrypt your data. This tends to be bad!
Full answer: Any scheme 'like' the One-Time-Pad (OTP) breaks down into two sections: first we form a pad and second we combine the pad with the plaintext. The second of these tends to be through the xor operation (denoted $\oplus$), but this doesn't really matter for this question.
So, the first section: generating a pad. For the OTP, we assume the pad is perfectly random and secret. If so, then the scheme is perfectly secure. However, this means we somehow have to transmit a large amount of secret information to the other person before we can send them the message, which in most cases just means we pass the difficult problem of key distribution upstream.
So, how do we get around this problem? We take a small key that can be more easily distributed, and expand it with a method that everyone involved knows to form a stream cipher. This is a function that tries to expand a small key into a suitably large key stream to pad the message with, and we have some pretty tough requirements of it. In particular, we demand that the key stream appears random. That is, someone seeing even a large amount of the keystream should not be able to differentiate it from truly random data (within 'reasonable' computation effort). If they cannot differentiate it from random data, we decide that the output ciphertext is sufficiently secure.
Before I try and pull things together, I'd like to point you to Kerckhoffs's principle. Basically, for any real level of security in the modern world, we require that a cryptographic algorithm should be secure even when the attacker knows everything about it except for the key. Note that 'the key' is a vague notion - one can draw the algorithm-key boundary pretty well wherever they like - but good practise says the key should be as small as possible (eg algorithm=RandomAccessMachineSimulator; key=MyCryptoAlgorithm) without compromising the security of the algorithm.
So, how does your situation work? Well, turning a large amount of not-very-random data into a small about of 'quite random' data is a question of [extracting the entropy][Note 5]. A naive way of doing it would be to simply pump all your input data through a key derivation function like PBKDF2, but I won't go into this in much detail chiefly because there's a bigger problem: What do you keep secret? - Keeping the entire algorithm secret: If you want to keep everything secret from an attacker, then how do you tell the person you're messaging how to decrypt it? Somehow you must privately give your ally all the required details. At this point though, why not just give them an AES key that you can then use to communicate securely? - The data-source is secret: This makes a bit more sense. All the 'fiddly' computer bits (eg the entropy extraction) can be exchanged insecurely (eg email), then you send them the data-source (URL?) in secret. However, (assuming that for some reason this time we can't just send a key instead) anyone listening to your HTTP requests will hear all URLs loaded, and by testing each one of them will find your key very efficiently.
[Note 5] - Is there already a good description of this somewhere?
What you've described is generally called a "book cipher" or "Ottendorf cipher", where the "key" is knowing which publication is being referenced, as well as the algorithm for recovering information from it.
A hundred years ago they were quite secure because not only were books fairly rare, but trying every book against an unknown cipher was very time consuming. Some historically famous codes, such as the Beale Ciphers, used this method, and they withstood a lot of very motivated attackers for a very long time.
But with the digitization of vast quantities of books and other print resources by huge organizations such as Google, (and security agencies) they are now pretty much a historical footnote.
I agree with John Deter's answer. If you attempt to access some website to get some public information from it, then your favourite intelligence agency is going to have a record of that due to their extensive spy network which has taps on every internet backbone. If two people access the same site then appear to be communicating with each other using OTP then an intelligence agency could eventually figure out the secret scheme based on the content of that website.
If you and your partner went to a shared computer (e.g. internet cafe, library) without your mobile phone/surveillance device and paid in cash then accessed some pre-agreed upon website and used the same secret key retrieval scheme then that would be less likely to be discovered by one of the intelligence agencies.
As for the key construction you need something random, or pretty close to it for a one-time pad. One idea might be to meet in person first (without your mobile surveillance device listening in) and agree upon a daily key extraction method from a public source. Perhaps both of you buy a physical copy of a newspaper each day with cash which was printed at the same time (same edition). This newspaper might be distributed geographically so you need one that was printed in the same run or things might have changed between editions. That can be distinguished by the numbers/fine print on the inside of the first page, bottom corner (this depends on the newspaper). Once you have the same newspaper you can work out a scheme to extract your key data. Obviously you would devise your own secret method, but you could pick a page number, or group of pages, find certain headlines, paragraphs or sections of content that are in the same area of the page from day to day. Then extract every 2nd (or nth) letter from that content. Once you've gathered say 64 characters you could reverse the order of them, then group them up and run it through a 512 bit cryptographic hash such as Keccac or Skein which will give you a pseudo random uniform distribution of random bits. That would give you a key unpredictable enough that no intelligence agency could ever guess it. From there you get your plaintext (and your key cut to the same length), convert them to binary and XOR them together, MAC the ciphertext with an information-theoretically secure MAC and send your ciphertext and MAC off to the other person via whatever communications method you're using.
This method might be good for a limited number of messages per day. Probably slightly time consuming to create the key each day too. The main problem with this method is it relies on a secret method to create the key. If either one of you is captured and interrogated by an intelligence agency then you might be forced to reveal your message extraction method, at which point your past communications would be compromised as they could dig up the old publications and recreate the keys. You could probably lie about it or say you randomly pre-created the keys when you met up one time but their extraction methods (waterboarding & sodium pentathol) are apparently pretty effective.
It is my understanding that a pre-agreed astronomical noise source with a protocol presenting a few sync bytes would do the job. Nobody records enough astronomical noise with telescopes to have a good chance of having recording your source.
However: if somebody happened to record it that day, you're sunk.
