27
$\begingroup$

I asked an AI the following question:

Can you provide me with random numbers of 30 digits in length?

And then the AI has generated these numbers for me.

563958422461839604397274590248
743298571529845197630149526734
962345019834590239458293827563
841259630492576302945836184025
129458392650293745092837563945
938475029835820943759284396284
284657492870476502834795836192
937453090283964506295830295830
293857692057983473950296395830
912465732491850173082913290227
107609347826170463295771141645
788604572134668302261094130853
302239005287747759752543651912
819043729526479834375692849203
503284901875216436897123740615
641375927104689705914038297506
474327512378904132524817120687
968370218489946036482905748215

Now I wonder:

  • From the point of view of information theory or cryptography, are these numbers random?
  • From an algorithmic point of view, how are these numbers predictable?

The AI in question is LLaMA from https://github.com/ggerganov/llama.cpp

$\endgroup$
14
  • 16
    $\begingroup$ When you say "I asked an AI", do you mean "I asked ChatGPT"? $\endgroup$
    – Stef
    Apr 25, 2023 at 8:39
  • 11
    $\begingroup$ For those coming from HNQ unfamiliar with randomness in cryptographic context: the random numbers must be generated in such way that no algorithm can predict the next digit, even given VERY long stream of what's already generated; also even when the seed is compromised, past random numbers cannot be recovered or deduced. These 2 requirements are roughly known as: prediction-resistance and back-tracing-resistance. $\endgroup$
    – DannyNiu
    Apr 25, 2023 at 9:15
  • 2
    $\begingroup$ @Stef, I use LLaMA from github.com/ggerganov/llama.cpp $\endgroup$
    – swannty
    Apr 25, 2023 at 9:25
  • 14
    $\begingroup$ That's not apparent from the title, and since you've not even indicated the AI in the question (initially) the answerers are not required to take above statement into account. How numbers are generated can usually not be concluded from looking at a few numbers. $\endgroup$
    – Maarten Bodewes
    Apr 25, 2023 at 9:32
  • 2
    $\begingroup$ @swannty Yes, answer by Joseph mentioned observing repeated pattern. That'll b a fatal wound on any PRNG in the cryptography context. $\endgroup$
    – DannyNiu
    Apr 25, 2023 at 9:39

10 Answers 10

49
$\begingroup$

First of all, you have only 540 digits, and 540 digits seems a bit small. I would have asked the AI to generate as many digits as it can. Nevertheless, I was able to find the following sources of non-randomness here by casual observation followed by some statistical analysis.

The string 295830 repeats twice on the end of the 8th row, and we see the related string 395830 on the end of the 9th row. The string 30294583 appears in one place while the similar string 3029583 appears in another place.

There are only 13 instances out of 539 where a digit repeats. There should be about 54 instances of a repeating digit. It seems like the AI avoids repeating digits because they may appear 'non-random' or because the AI learned incorrectly from humans that random strings do not repeat characters very much.

We can actually calculate the probability of having at most 13 repeated digits out of 539 instances if the string were truly random since this distribution is a binomial distribution. Here, the probability of getting $k$ instances out of 539 instances of a repeated digit is $\binom{539}{k}\cdot 0.1^{k}\cdot 0.9^{539-k}$. Therefore, the probability of getting at most 13 out of 539 repeated digits is $\sum_{k=0}^{13}\binom{539}{k}\cdot 0.1^{k}\cdot 0.9^{539-k}\approx 4.90903\cdot 10^{-12}$. We can therefore safely conclude that these digits are not random/pseudorandom.

My word embedding algorithm detecting non-randomness

So I fed these 540 digits into my own complex matrix-valued word embedding algorithm with dimension $d=30$. I can tell that the word embedding converges to a good local maximum for two reasons (the word embedding does not always converge to a good local maximum):

  1. I trained the word embedding twice, and I got the same fitness level both times (with $1.31\cdot 10^{-14}$ difference in fitness).

  2. Even though the matrices were complex during the initialization and training, after the word embedding was fully trained, I checked that the matrices were real up-to-symmetry but computing the traces of their products. If I tried training using real matrices though the word embedding may converge to a suboptimal level of fitness (but it did not in this case which is good), so I needed to use complex matrices during training; the fitness function has a lot of near singularities and complex matrices are good at avoiding these near singularities while real matrices are not.

  3. I trained the word embedding using quaternionic and real matrices and the trained word embedding was the same thing (up-to-a quaternionic unitary matrix or real orthogonal equivalence) as I got when training with the complex matrices.

After training, I got a fitness of about -1.1372531160707808. A smaller fitness value after training means that the sequence is more random while a larger fitness value means that the sequence is less random. I have been searching for a random sequence that has a higher fitness than the fitness for the AI generated sequence, and I have not been able to find such a random sequence.

I call my word embedding algorithm an MPO word embedding which stands for matrix product optimized. Suppose that $A$ is a set. In our case, $A$ will be the set of digits from 0 to 9, but in natural language processing, $A$ will be the set of tokens. Let $a_1\dots a_n\in A^*$ be a string.

Suppose now that $f:A\rightarrow M_d(\mathbb{C})$ is a function. Then define the fitness $$L(a_1\dots a_n,f)=\log(\rho(f(a_1)\dots f(a_n))^{1/n})-\log(\|\sum_{a\in A}f(a)f(a)^*\|_p)/2.$$ Here, $\|X\|_p$ denotes the Schatten $p$-norm of a matrix $X$ while $\rho(X)$ denotes its spectral radius. We maximize the fitness using gradient ascent. In our case, $p=2$, so $\|X\|_p$ is just the Frobenius norm. One reason why we get the same fitness after training multiple times is that the dominant eigenvalue of $\sum_{a\in A}f(a)f(a)^*$ is substantially larger than the other eigenvalues which means that there is a rank-1 positive semidefinite matrix that somewhat approximates each $f(a)$. In other words, we are making a tradeoff between being able to use all $d=30$ complex dimensions equally and always converging to the same local optimum. The matrices $f(a)$ were also approximately low rank matrices. If the matrices $f(a)$ were all rank-1 matrices, then the MPO word embedding would reduce to a weighted digraph embedding where the weight of $(u,v)$ is the number of times the string $uv$ occurs in the cycle of 540 digits.

Here is a more detailed description of my word-embedding algorithm, and here is a link for code on my Github account.

$\endgroup$
16
  • 11
    $\begingroup$ Well spotted. The AI may be also be a chatbot that tries to generate random numbers from previously seen values. That's generally the way they work after all. $\endgroup$
    – Maarten Bodewes
    Apr 24, 2023 at 23:25
  • 3
    $\begingroup$ return 7 chosen from fair dice roll. $\endgroup$
    – Aron
    Apr 26, 2023 at 7:58
  • 7
    $\begingroup$ @Aron Surely you mean return 4? $\endgroup$ Apr 26, 2023 at 14:00
  • 1
    $\begingroup$ So... amazingly it generates random numbers about as poorly as a human. Or maybe a little better $\endgroup$
    – Cruncher
    Apr 26, 2023 at 16:20
  • 2
    $\begingroup$ @JimmyJames: So more accurately we might say the LLM has learned that humans think "random" numbers are biased away from repeating digits. "Afraid" can work as short-hand for that idea, but fair point that it could be misleading for readers who don't understand the intended meaning. $\endgroup$ Apr 27, 2023 at 3:29
11
$\begingroup$

Answering the title - yes, it can.

A neural network can generate random output if it contains a layer introducing randomness. For example - when I use the TensorFlow framework I can consider tf.random.uniform function and put it inside the model. It will cause that "random" numbers will appear inside and the output will contain randomness, as well.

$\endgroup$
3
  • 10
    $\begingroup$ This answer is okay, but not satisfactory in the context of cryptography. See my comment under OP. $\endgroup$
    – DannyNiu
    Apr 25, 2023 at 9:16
  • 3
    $\begingroup$ @DannyNiu I don't see why an AI couldn't have access to source of true randomness that is sufficient for cryptographic use. Whether it can actually use it appropriately is a different question. $\endgroup$ Apr 27, 2023 at 14:11
  • $\begingroup$ I'm not sure how reasonable this is an as answer. This seems more like an answer for "if I put random numbers into a neural network, can they still be random when it comes out", since I wouldn't really count tf.random as being "part" of the neural network. The detail of "part" is pedantic, but its important because I think the much more interesting question is "can a neural network generate 'random' numbers without a random source" and the answer is probably no $\endgroup$
    – BeB00
    Apr 27, 2023 at 22:42
9
$\begingroup$

That will depend on the AI:

  • Currently it is likely that chatbots piece the random values together from previously observed values, which of course would make the result mostly non-random.
  • An AI may also request the random values from a separate API or service; AI's are using external services at a very high pace.
  • I'd say that it is currently unlikely that it would try and generate the random values itself (i.e. by using its neural network, in the same way as humans do). If it does then remember that humans are terrible when it comes to generating random values.

If it is using an API then it is unclear if the random number generator is secure and well seeded. Both are requirements for any cryptographically secure pseudo-random number generator. Nor is it clear if the values are obtained from an external service or if they are stored anywhere. As such, I would definitely not trust any the obtained values to be secure out of the box.

Note that it is impossible for anybody to tell you if these values are secure without more input. It might be possible to spot vulnerabilities, but you'd need to analyze the entropy source / true RNG and the PRNG algorithm to get anywhere "proving" security for these values.

It might be a good idea to ask the AI how it generates random numbers and then to verify the answer. If it doesn't integrate with an API to provide random values then the chances of getting cryptographically secure random numbers is so small you may safely ignore it.

$\endgroup$
3
  • 3
    $\begingroup$ I'm less optimistic than you about "It will probably request the random values from a separate API or service. I'd say that it is unlikely that it would try and generate the random values itself (i.e. by using its neural network)." AIs like Siri or Alexa are designed to only act as an interface, and call on external APIs, but a text-generating AI like ChatGPT or LLaMA is designed to produce its own text all on its own, which is why they are so bad at answering requests that are too logical and algorithmic. $\endgroup$
    – Stef
    Apr 25, 2023 at 9:48
  • $\begingroup$ A GPT model could emit equal weights for digits 0,1,2,...,9 and then the next token selection part of the algorithm (which selects tokens randomly according to the weights given by the model) would select truly random (or truly pseudorandom) digits. $\endgroup$
    – user253751
    Apr 25, 2023 at 14:44
  • 1
    $\begingroup$ Yeah right, well, without any proof that it does so, and the knowledge that ChatGPT is often wrong regarding statements even about itself, I'm not convinced that it can ever generate random values that way. It would be hard for it to prove that it would be secure anyway. $\endgroup$
    – Maarten Bodewes
    Apr 25, 2023 at 15:45
7
$\begingroup$

This question can be answered on a couple of levels.

First, on the level of basic intuitions, these AI systems are trained to imitate human text. We would therefore on a first order approximation expect them to perform similarly to unaided humans when generating pseudo-random numbers. Unaided humans are terrible at generating random numbers.

Second, we can (at least in the case of some AIs) successfully ask the AI itself. When I ask ChatGPT3.5 to "Write Python code that can run some common statistical tests on (this sequence of digits)", it suggests calculating chi-squared goodness of fit tests on the observed digit stream against the null hypothesis of even distribution for single digits and digit pairs. Eyeballing the code, the code seems largely correct (with a minor off-by-one error miscalculating the expected frequency of each pair, which should be totally inconsequential for results). When prompted accordingly, it admits this mistake and suggests a correction which looks good, but errors out. After having it fix that error, the code looks correct and produces the following output

Chi-squared test:
Chi-squared statistic: 15.26
P-value: 0.0841
Chi-squared test (digit pairs):
Chi-squared statistic: 190.71
P-value: 0.0000

So, digit-pairs look highly non-random.

Thirdly, the AI itself roughly speaking pseudo-randomly samples a model of human text generation. What RNG is used here depends on implementation choices. It is perfectly possible (and would not add significant computational work to the generation process of any reasonably general text-generation models) to use cryptographically secure random numbers here, but I would strongly expect that this will not be the case in practice (although maybe watermarking schemes for AI text could be a reason to use a cryptographic source of randomness in the sampling strategy). As such, the text generator only acts as an obfuscation layer over an insecure RNG here. At that, it is obviously an obfuscation layer that introduces statistical nonuniformity instead of removing it.

Fourth, if one asks e.g. ChatGPT3.5 for a way to generate cryptographically secure random numbers in Python, it outputs code using Python's secrets library. This is a standard library designed for generating cryptographic random numbers, and I would expect it to be a reasonable choice in many settings (although I haven't looked at its implementation choices). It is certainly by light-years more reasonable an approach than asking a neural network to do it.

$\endgroup$
2
  • 4
    $\begingroup$ When humans need to include 340 digits in their texts, I assume they're typically not unaided. But I'd be more afraid of the AI copying digits from not-random-at-all lists of digits. For instance, there must be more texts containing digits of pi than texts containing any other long sequence of digits; hence I would be afraid that an AI would be biased towards the digits of pi every time it's asked for sequences of digits. $\endgroup$
    – Stef
    Apr 25, 2023 at 9:41
  • 1
    $\begingroup$ The "underlying PRNG" is not the numpy PRNG. It's the PRNG used by the "driver" program for the model (see the temperature setting), passed through the model itself (which will tend to heavily bias the output sequence; that is, after all, its purpose). $\endgroup$
    – wizzwizz4
    Apr 25, 2023 at 16:48
6
$\begingroup$

Like any other RNGs based on deterministic algorithms, an AI will not be able to generate real randomness. Whether it meets the required level of randomness depends entirely on how it generates the numbers. But without any randomness as input into the AI, it will definitely be NOT RANDOM. Anyhow, since AI is usually easy to manipulate, it is not a good idea to use those random numbers in any cryptographic context.

AI only appears intelligent because its algorithms are too complex for a human to comprehend. That it is therefore capable of generating a safe random value is a fallacy.

Edit: I think the discussion in the comments is all due to different definitions of AI. Of course there are AI's that use real randomness as input and may make use of it. What I meant is that current approaches such as Machine Learning / Neural Networks are not able to generate randomness but still might make use of it from inputs it gets.

If the RNG is included in the "AI" definition, then AI may generate randomness. If the RNG is exluded from the definition, the AI is not able to generate any randomness, but still may output randmoness based on random inputs it gets. In the second case, the AI would be deterministic and the output would be predictable, when knowing the inputs into the AI. In the first definition case, it would not be deterministic.

$\endgroup$
10
  • 1
    $\begingroup$ "AI only appears intelligent because its algorithms are too complex for a human to comprehend. " Actually, the algorithms behind GPT2 are surprisingly simple. It's kind of a surprise that the output is as good as it is. $\endgroup$
    – Stef
    Apr 25, 2023 at 9:44
  • 2
    $\begingroup$ That's correct. Actually, I meant less the complexity of the algorithms (which are usually written by humans anyway) and more the complexity of understanding how a trained neural network maps the inputs to the outputs. Keyword "Explainable AI". $\endgroup$ Apr 25, 2023 at 11:38
  • 7
    $\begingroup$ There's no reason an AI couldn't have access to a source of entropy used to generate true random numbers. $\endgroup$ Apr 25, 2023 at 12:17
  • 3
    $\begingroup$ GPT is not based on deterministic algorithms. There is an essential part which uses randomness. $\endgroup$
    – user253751
    Apr 25, 2023 at 16:31
  • 1
    $\begingroup$ @user253751 But it does not generate the randomness, does it? To say that AI is able to produce randomness by using a randomness source as input does not make any sence to me (in terms of this question). An AI itself is deterministic. It might use randomness to produce random locking outputs. But this randomness is taken from any source and not created by the AI. In general, no computational algorithm is able to create randomness, it always has to use some kind of randomness source. $\endgroup$ Apr 25, 2023 at 17:19
4
$\begingroup$

The correct measure of randomness is entropy. Entropy is defined for probability distributions, e.g. a fair coin toss has more entropy than a biased one. It makes no sense to say "how random is this number" without knowing with what probability it is produced. For all we know, the AI might always produce the same strings in the same order when asked to provide randomness. Like another answer mentioned, a deterministic algorithm cannot produce randomness. The best you can hope for is to apply a statistical test to the string to test the hypothesis that it was generated from a uniform distribution.

On a side note, asking an AI for randomness seems overkill, much like opening a coconut using a bazooka.

$\endgroup$
2
$\begingroup$

In general, a GPT AI architecture is capable of generating properly random numbers. That's because the overall design consists of two stages: a trained neural network which gives probabilities for the next possible tokens in the sequence, and then a simple piece of code which picks one at random based on those probabilities.

Due to the second stage, this system design intrinsically has plenty of randomness available to use. Indeed, every single token is the result of a random choice.

If the neural network was perfect, it would give the same probabilities for each of the digits 0-9, and then the random stage would pick one completely at random (subject to the quality of the randomness available to the system).

In practice, the neural network probably does not output exactly the same probabilities for the digits 0-9. But it could, so the answer to the title question is "yes."

$\endgroup$
2
$\begingroup$

Maybe it generates numbers at random the way a person does: it is well known that people who think they are generating random numbers actually introduce all sorts of biases. For instance, a person may conscientiously make sure that all ten digits have equal frequencies, but a sequence that is generated by a truly random process, such as Ernie, frequently violates this.

$\endgroup$
1
$\begingroup$

Randomness is best understood, IMHO, as relative to information at hand. Perhaps there's true randomness at the quantum level, but at the macro-scale it's an illusion of ignorance.

For example, if I know all of the forces exerted on a coin when it's flipped, while it's in the air, and the properties of the surface where it will land, I could (conceivably) predict the outcome. Similarly, if I know the current state and and PRNG algorithm, I can predict the next number generated by a pseudo-random number generator (PRNG). If I know all about my CPU and its usage I could even predict the behavior of a heat/cpu based RNG.

The real issue with PRNG is not their lack of randomness (relative to the user's knowledgebase), but the fact that the numbers lack independence between draws (hence our ability to generate a repeatable series of number using a particular seed).

So even if an AI uses a PRNG, from the perspective of a user the number generated is random. The second one, if generated from the first, not so much...

$\endgroup$
0
$\begingroup$

While there are all sorts of limitations on AI systems being able to generate random numbers, as discussed in other answers (especially if a system is just regurgitating other content), an AI system should in theory be able to read values from an external source of effective randomness (including already-connected sensors) and report those values. Whether or not any particular AI system does that depends at least in part on its design.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.