This is a soft question regarding comparisons between government security services (eg, NSA or GCHQ) and open-source research (e.g., academia). Hopefully it's on-topic for this site!

In essence, my question is the following.

How far ahead (if at all) of of open source (specifically academic) research are governmental agencies?

There are some various thoughts I've had on this matter.

  • NSA is the largest employer of mathematicians worldwide, and their primary focus is on security. So not only do they have an incredibly large number of people, but they're also very focused. On the other hand, cryptography departments at universities (that I'm aware of) don't seem to be particularly large.

  • We now get onto the actual people who do the work. For example, to start at the base level at GCHQ you need a 2:1 in a three-year maths degree. (I'm from the UK, and don't really know any comparison with American GPA.) In comparison with university faculties, these will all have doctorates as a minimum, and university researcher places (at least in UK universities I'm aware of) are very competitive.

  • At a first glance, this suggests that the standard of researchers in general is much higher in academia. Of course, one has to be careful: not everyone wants to do a PhD for starters; moreover, a lot of people may not want the job-insecurity of post-docs, and prefer to move into a more secure (likely higher-paying) job outside of academia. Furthermore, continuing from the first point, these people then spend all their time working on this stuff, with large amounts of resources (including many other people around).

  • As an example, when DES was replaced with AES, instead of developing AES themselves, the NIST set a challenge to other researchers. The NIST then chose the best of these (supposedly, anyway; many people have apparently legitimate claims that the most secure wasn't chosen, but instead the one the the NIST 'liked' the best).

  • I have heard that RSA was discovered some 8 (or maybe 3?) years before it was public, but that this was at a time when there was comparatively little open source research into such topics.

  • Finally, a lot of what secret agencies do is, well, secret!

Basically, there seem to be arguments for many points, and I was wondering if anyone has more insight than I do!

Very related (but not the same) is this question: How can we reason about the cryptographic capabilities of code-breaking agencies like the NSA or GCHQ?; that question is more about some specific things (eg symmetric ciphers).

To be clear, I know this question isn't definitively answerable without high-up knowledge of governmental affairs, but I feel some approximations should be possible.

Of course, none of this addresses private companies (eg, Google). These employ a lot of people, likely(?) with higher entry requirements than NSA/GCHQ (see above on why that doesn't give a clear cut implication) and also have lots of money; moreover, there's likely(?) less bureaucracy.

If any of my above claims are wrong, please do point them out! I'm not very knowledgeable in the world of cryptography. I'm a maths PhD (discrete probability), but have interest in this stuff.

  • 3
    $\begingroup$ NIST chose AES, not the NSA. From what I heard, the NSA preferred RC6 over Rijndael. $\endgroup$ Aug 20 '17 at 10:53
  • $\begingroup$ Ah, my mistake! Let me edit that :) $\endgroup$
    – Sam OT
    Aug 20 '17 at 10:54
  • 2
    $\begingroup$ Obligatory. reactiongifs.com/r/clssfd.gif $\endgroup$
    – fkraiem
    Aug 20 '17 at 11:13
  • $\begingroup$ Is "open source" meant as a synonym of "academic" or "scholarly" here? $\endgroup$ Aug 20 '17 at 21:17
  • $\begingroup$ @TobiaTesan I would assume so. Open Source "research" is just research whose conclusions, data... is available to be built upon by anyone who wishes. I'm thinking that the OP means "publicly accessible" instead of open source. $\endgroup$
    – Zizouz212
    Aug 21 '17 at 6:19

Disclaimer: This post is possibly opinion based.

How far ahead (if at all) are governmental agencies of open source (specifically academic) research?

This question is impossible to answer. By definition, this require a knowledge of the state of research in such agencies which is definitively something they want to be kept as secret...

thx to @fkraiem:


However one can get a feeling of it by considering the followings:

  • Academic research is usually founded by companies or government agencies.

    e.g. for the Gimli paper:

    This work was supported in part by the Commission of the European Communities through the Horizon 2020 program under project number 645622 (PQCRYPTO) and project number 645421 (ECRYPT-CSA); the Austrian Science Fund (FWF) under grant P26494-N15; the ARC project NANOSEC; the Belgian Fund for Scientific Research (FNRS-F.R.S.); the Technology Foundation STW (project 13499 TYPHOON), from the Dutch government; the Netherlands Organisation for Scientific Research (NWO) under grant 639.073.005; and the U.S. National Science Foundation under grant 1314919. “Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

  • In the case of important projects, academic research is paired with companies with huge computing power (Google, Microsoft Research...) e.g. SHAttered:

    This result is the product of a long term collaboration between the Cryptology Group at Centrum Wiskunde & Informatica (CWI) - the national research institute for mathematics and computer science in the Netherlands - and the Google Research Security, Privacy and Anti-abuse Group. Two years ago Marc Stevens and Elie Bursztein, who leads the Google's anti-abuse research team, began collaborating on making Marc's cryptanalytic attacks against SHA-1 practical by leveraging Google expertise and infrastructure.

    Sure such companies may? still not have the computing power of NSA, but the minds in there are definitively on par with such agencies.

I would also advise this to have a look at this: The Moral Character of Cryptographic Work by Phillip Rogaway [paper] [Usenix Talk], specially at the founding part (p. 36). We see a increase in the number of project founded by military agencies.

WARNING: this is a personal assumption, you may chose to disagree with it.

Knowing their missions (cf quote bellow) we can somewhat deduce that their level of research is most likely on par with academic work (why would you heavily fund something that would be bellow your current level of comprehension?)

e.g. for Defense Advanced Research Projects Agency (DARPA) :

The genesis of that mission and of DARPA itself dates to the launch of Sputnik in 1957, and a commitment by the United States that, from that time forward, it would be the initiator and not the victim of strategic technological surprises. source.

Their goal are the following:

  • fund academic research
  • but most importantly know what you are currently researching on! $\Leftarrow$ This is information they want!

TL;DR: I believe they are on par with current academic research.

  • $\begingroup$ Thanks for your answer! What you're saying is approximately what I thought was likely the case. I agree it's unlikely they'd be behind (and indeed by definition of open source anything published from universities they'd have access to, so by one definition couldn't be "behind"). Some claim NSA are years and years ahead of open source research, but to me this seems unlikely... \\ I have seen Rogaway's paper before, including the funding; I've heard Assange talk on such topics also. $\endgroup$
    – Sam OT
    Aug 20 '17 at 11:43
  • 6
    $\begingroup$ It almost doesn't matter any more to try and distinguish between government and private industry (or academia) because many companies ( you can fill in the blanks here) are just fronts for the government. And many so called "security experts" are secretly working for the government as well, yes folks, we live in a very complex world ! $\endgroup$ Aug 20 '17 at 14:02
  • $\begingroup$ Do you really need to include a 3.4 MB image to say "it's classified", something which can be accomplished in a few dozen bytes even if one counts HTML markup? $\endgroup$
    – user
    Aug 21 '17 at 13:33
  • 1
    $\begingroup$ @MichaelKjörling It gave me a chuckle, so I think it was worth it. $\endgroup$
    – forest
    Sep 4 '18 at 3:02

Fundamentally, I do not believe you can compare them specifically because they tend to have different behaviors. I am an academic who does the non-classified work in conjunction with government teams; however, I'm not a cryptographer, but a circuit designer. I went from industry back to academia as a postdoc just to work in this space. I also don't have a security clearance, so I can specifically publish even if most of it doesn't get out.

In academia, things tend to lean toward an idea more than an application. In the application space, we have constraints and often no pricing pressures so we aren't bound to the hardware. Academic cryptographers tend to prefer complex round functions whereas the government teams usually spent time aggressively simplifying things for the constraint sets. (you see this tradeoff when see a simpler round function with more rounds) Things that the government doesn't think is a problem but academics love: "memory hard function". I can put 8GiB of RAM with through vias on the back of a POWER core on 14nm SOI. This means my IO is 10x faster at a minimum and no cache misses (I don't even put cache on dies in this case). Academic cryptographers generally are not making custom ICs, but there are exceptions.

I'd basically sum it up as academics are ahead with their constraints, and the government is ahead with their constraints.

  • $\begingroup$ Interesting take, +1! To be honest, I don't understand what a lot of the stuff in your second paragraph means, but I get, and agree with, the general idea of academics being more idea-focused and governments being more towards applications. What do you think about comparing with technological companies who are also pushing for applications (to then become consumer products, albeit specialised)? $\endgroup$
    – Sam OT
    Aug 20 '17 at 15:34
  • 1
    $\begingroup$ @SamT Companies just push what they already sell. ARM is a good example, as if you read the NIST 8114 feedback on the upcoming lightweight crypto spec, they are completely daft to why someone wouldn't want to use an ARM in their system. (I never use ARMs as I have better cores for area/power/throughput and no IP burdens) Companies generally push specs around products that exist. Also, I'll add links to the above in an edit. $\endgroup$
    – b degnan
    Aug 20 '17 at 17:15

From a slightly more esoteric perspective, I'd be surprised if governments are much further ahead of the public these days. As we all pass through time, we leave our fingerprints on our surroundings. Entropy inevitably increases and I believe that eventually all truth will out. It can not be contained for ever. That's why it's very difficult to hide a crime once it's discovered. There's some appropriate saying about all the people, all of the time. This isn't an opinion, it's thermodynamics.

If you read around, there is lot of detail as to what governments get up to. And that information is then propagated, so if GCHQ knows what Pyongyang knows, the Sun newspaper will know it soon too. Secrets sell copy. I'm highly cynical, but part of that cynicism is that most people want money, air time or to do the right thing.

If governments are doing something bad, that tends to leak sooner than if they just go about their national security business in ethical ways. There will always be Assanges and Snowdons. A flash drive can hold many pages of documents and they can easily be published to the entire planet. Simply posting on USENET is virtually irrevocable. That's why I know that we did go to the moon and that the CIA did not kill Kennedy. Someone would simply admit it. I'm not sure about Elvis being dead though...

  • $\begingroup$ As you say, you do sound highly cynical! =P -- but you do make some good points. Particularly to do with large-scale conspiracy theories, which require hundreds of people to not ever leak anything. However, the second law of thermodynamics needs to be applied carefully: it doesn't say anything about time-scales on which stuff happens. I do feel that it's likely that governments have some interesting stuff hidden away... just not as much as they'd like us to believe! $\endgroup$
    – Sam OT
    Aug 20 '17 at 16:45
  • 1
    $\begingroup$ Really Paul, you believe in the Warren Commission ? LOL ! How does Allen Dulles,the head of the CIA who was FIRED by President Kennedy in 1961, wind up heading the investigation into his murder??? Nothing wrong with that right, makes sense to me ;-) $\endgroup$ Aug 21 '17 at 4:51
  • $\begingroup$ @WilliamHird Of course not - Dulles worked for Moscow. The point I'm making is that even if he did, increasing entropy means that it would have come out after all this time. I'm 81 and have stage 3 cancer. I go to the Post and say give me $1M for my grand kids and I'll give you these documents exactly detailing the operation I ran to kill Kennedy... This happens all the time. $\endgroup$
    – Paul Uszak
    Aug 21 '17 at 11:49
  • $\begingroup$ @PaulUszak: Remember the line from the movie "Three Days of the Condor " where Robert Redford tells Cliff Robertson that he (Redford) had given his story to the N.Y. Times and Robertson says to him "what makes you think they are going to publish it? ". $\endgroup$ Aug 21 '17 at 13:02

Not the answer you're looking for? Browse other questions tagged or ask your own question.