People in the AI space, myself included, seem to typically believe that the smartest models that exist today were trained by scaling labs.
But I have met non-AI people who believe that the US government likely has access to secret models more powerful than the labs’.
What is the best evidence either way? If (as I expect) the current evidence points towards today’s smartest models having been trained by labs, what evidence should I look out for in the future that would indicate that the situation had changed?
People get a thrill from telling stories like that. Invariably they heard it from this guy who heard it from this guy who can’t say where he heard it, but believe me...
The probability that any government entity in the world has independently developed anything even close to what the “labs” have is very small. It’s not easy to hide that kind of compute, nor is it easy to get the necessary talent. The government can’t spend a trillion dollars without anybody noticing, nor build giant data centers or huge state-of-the-art fabs.
Maybe they’re ahead in very narrow areas directly related to their jobs: things like small models for specific military applications, or versions of the “labs”′ LLMs fine-tuned for breaking security. But I doubt even that at this point.
There is a really good chance that they have access to “lab” models that normal people don’t. Less restricted, and less enstupidified by “safety” training. But that’s not the same as being ahead of the “labs”.
They’re probably not particularly far ahead in cryptography nowadays either. But at least cryptography is something the NSA actually had and has as a mission. They had by far the most resources on it, and first pick of the talent, and a big head start, and some hybrid of a gentlemen’s agreement and a coercive arrangement that everything anybody outside invented was disclosed to them and nobody else, for a long time… in the last century. It was specialized military technology then. Now it’s dual use, or indeed just plain general-purpose stuff. And no, they cannot break any of the ciphers in wide use now, assuming they’re used correctly (which they’ve sometimes tried to undermine, but at best managed to delay).
In 1960 or 1970, the US Government, or at least the military arms, had a pretty large lead over everybody in all forms of “high tech”. They got everything first and best. That hasn’t been true for a long time.
I had a somewhat heated IRL argument a while back with a smart-but-not-AI-knowledgeable person who insisted that the U.S. government had secret models far beyond what any private company had. My key points against his claim were:
All major AI developments so far have stemmed from publicly-available papers. DALL-E, for all the initial speculation, was developed based on image diffusion models. Likewise, there have been open-source multi-modal transformers since long before GPT-4o came out. To outpace publicly available research by any significant margin, you’d need to simultaneously outrun the combined academic and private sector industrial base of not just America, but the entire global research community.
(This differs from just knowing a few tricks of the trade to get slightly more out of existing paradigms, which is rarely the claim being made by these sorts)
Successful AI labs have world-famous talent behind them. To carry out an AI “Manhattan Project”, you’d need to find people who have spent decades creating AI research papers with thousands of citations, and arrange things such that absolutely nobody wondered why they fell off of the public radar. This is vastly harder in the age of social media.
Keeping major secrets in a relatively tight-knit research community is extraordinarily difficult. Most people are in it for the love of the game, and will end up talking to their friends at DeepMind, or OpenAI, or Anthropic, or XAI about exciting developments.
To answer your secondary question, if I were looking for evidence that the situation had changed, I think I would expect the government to be relatively open about it, since a major public push would be the only way to attract a critical mass of talent. Even if this were not the case, big-name professors and scientists suddenly going silent or pursuing other opportunities would be the big indicator.
Seems unlikely, as others point out. The prior that the USG may have better versions of things than industry or the general public do is reasonable, and in some cases is born out (the NSA presumably still has a large lead in crypto and sigint capabilities, for example), but for the USG to have better models than the labs would either require that they be getting those better models from the labs and then not allowing the labs to keep using them or to have hired a large cadre of “dark” AI/ML engineers who have been doing secret research for months-to-years that is beating what the public knows (again, not a totally unreasonable prior given the NSA, but we don’t really even have rumors suggesting that this is going on).
There’s obviously no truth to that claim. Labs absolutely have better models.