Should we expect the PRC to support open weight models indefinitely? Or will incentives change such that Chinese frontier labs are forced to keep their weights secret?
It’s still too early to tell whether GLM 5.2 might have a “Deepseek” moment. Observers have become skeptical of benchmaxed open weight models that feel under performant in practice. However, the recent benchmarks are impressive, and we should have more “vibes” reports over the next couple days.
The releases comes on the back of Fable’s problematic safeguards and a subsequent executive order taking the model offline. Both events led to capabilities researchers emphasizing the importance of open weight models. So GLM 5.2 is well positioned to capture the attention of a wary research community.
Conversely, might this not also mean that the PRC is cued in to the risks of a model which claims to reach capabilities just behind Fable?
Open weight models seem to offer two advantages for the Chinese government. They allow Chinese labs to compete for consumers resistant to American proprietary models (e.g. anyone resistant to Anthropic API fees), and (more importantly?) they allow China to extend influence over countries which feel strategically disadvantaged by America proprietary models.
This is the problem of “mid-tier” powers, unable to compete with China and the US, uncertain of how to situate themselves in the AI race. The recent shutdown of Fable raises questions for how erstwhile American allies can ensure access to frontier intelligence. But if France can just use an open weight model from China, maybe they don’t need a fictional “le gros chaton” (assuming they are comfortable asking no questions about Tiananmen Square)?
However, if Chinese labs are entering into the same supposedly dangerous territory as Mythos, will China remain comfortable leaving these capabilities freely available to potential adversaries? They will draw their own conclusions from the US government’s erratic but wary approach to Mythos. And open weight models seem to have an inherently higher risk of jailbreaking and subterfuge. Is it consistent with the history of the PRC to assume they will want to give their populace greater access toward unregulated intelligence?
Allowing frontier labs to pursue open weight models has been advantageous to China until now, but I would anticipate incentives will change in the coming year, such that China implements industrial policy forbidding open weights for at least some subset of frontier intelligence.
I’m not sure if this would be good or bad for AI risk. Increased secrecy seems dangerous, but highly capable open weight models are perhaps more so?
I think “underdog” AI labs make their models open source because if they didn’t, no one will care about them, since everyone flocks to using the most competent models.
The Chinese government have cracked down on so many other things that it won’t surprise me at all if they ban open weights models. My guess is that right now, they feel no incentive because their models aren’t competing with frontier labs, and haven’t caused any tangible damage to them in any way. I agree that models like GLM 5.2 could change the equation.
If you’re submitting fiction or poetry to literary magazines, you need to be prepared to submit each piece a (surprisingly?) high number of times before you should consider reworking or retiring it, particularly if you only submit to top-tier publications (acceptance rate ≤1%). I think 20 submissions is probably the sweet spot.
Consider a simplified example.
Let’s say you submit to magazines with a 1% acceptance rate. Out of an audience of 10,000 writers, a random 2000 submit manuscripts, including yourself.
The editors have various biases (taste, fatigue, etc), meaning they cannot perfectly select the top 1% of submissions. Instead, they randomly select from the top 10%, meaning they select 20 of the top 200 submissions.
If you are not selected, what is the probability your manuscript is in the top 1%?
Before submission, if you assume no priors about the quality of your submission, we should assume a 1% probability we’re in the top 1%. By Byes law, after 1 rejection we should only lower that probability to 0.91%. In other words, we could say we’re still 91% sure our work is in the top 1% of possible works.
After 10 submissions, this drops to 0.37% (still pretty high!). After 20, we’re down to 0.13%. At 30, we’re down to 0.04%.
20 submissions feels like a good inflection point to me. 13% confidence is probably an underestimate, given extenuating factors like normativity of taste. Beyond this, unless you really believe in your writing, the opportunity cost alone isn’t worth the effort.
Should we expect the PRC to support open weight models indefinitely? Or will incentives change such that Chinese frontier labs are forced to keep their weights secret?
It’s still too early to tell whether GLM 5.2 might have a “Deepseek” moment. Observers have become skeptical of benchmaxed open weight models that feel under performant in practice. However, the recent benchmarks are impressive, and we should have more “vibes” reports over the next couple days.
The releases comes on the back of Fable’s problematic safeguards and a subsequent executive order taking the model offline. Both events led to capabilities researchers emphasizing the importance of open weight models. So GLM 5.2 is well positioned to capture the attention of a wary research community.
Conversely, might this not also mean that the PRC is cued in to the risks of a model which claims to reach capabilities just behind Fable?
Open weight models seem to offer two advantages for the Chinese government. They allow Chinese labs to compete for consumers resistant to American proprietary models (e.g. anyone resistant to Anthropic API fees), and (more importantly?) they allow China to extend influence over countries which feel strategically disadvantaged by America proprietary models.
This is the problem of “mid-tier” powers, unable to compete with China and the US, uncertain of how to situate themselves in the AI race. The recent shutdown of Fable raises questions for how erstwhile American allies can ensure access to frontier intelligence. But if France can just use an open weight model from China, maybe they don’t need a fictional “le gros chaton” (assuming they are comfortable asking no questions about Tiananmen Square)?
However, if Chinese labs are entering into the same supposedly dangerous territory as Mythos, will China remain comfortable leaving these capabilities freely available to potential adversaries? They will draw their own conclusions from the US government’s erratic but wary approach to Mythos. And open weight models seem to have an inherently higher risk of jailbreaking and subterfuge. Is it consistent with the history of the PRC to assume they will want to give their populace greater access toward unregulated intelligence?
Allowing frontier labs to pursue open weight models has been advantageous to China until now, but I would anticipate incentives will change in the coming year, such that China implements industrial policy forbidding open weights for at least some subset of frontier intelligence.
I’m not sure if this would be good or bad for AI risk. Increased secrecy seems dangerous, but highly capable open weight models are perhaps more so?
This reminds me of Alvin Anestrand’s Rogue Replication scenario...
I think “underdog” AI labs make their models open source because if they didn’t, no one will care about them, since everyone flocks to using the most competent models.
The Chinese government have cracked down on so many other things that it won’t surprise me at all if they ban open weights models. My guess is that right now, they feel no incentive because their models aren’t competing with frontier labs, and haven’t caused any tangible damage to them in any way. I agree that models like GLM 5.2 could change the equation.
If you’re submitting fiction or poetry to literary magazines, you need to be prepared to submit each piece a (surprisingly?) high number of times before you should consider reworking or retiring it, particularly if you only submit to top-tier publications (acceptance rate ≤1%). I think 20 submissions is probably the sweet spot.
Consider a simplified example.
Let’s say you submit to magazines with a 1% acceptance rate. Out of an audience of 10,000 writers, a random 2000 submit manuscripts, including yourself.
The editors have various biases (taste, fatigue, etc), meaning they cannot perfectly select the top 1% of submissions. Instead, they randomly select from the top 10%, meaning they select 20 of the top 200 submissions.
If you are not selected, what is the probability your manuscript is in the top 1%?
Before submission, if you assume no priors about the quality of your submission, we should assume a 1% probability we’re in the top 1%. By Byes law, after 1 rejection we should only lower that probability to 0.91%. In other words, we could say we’re still 91% sure our work is in the top 1% of possible works.
After 10 submissions, this drops to 0.37% (still pretty high!). After 20, we’re down to 0.13%. At 30, we’re down to 0.04%.
20 submissions feels like a good inflection point to me. 13% confidence is probably an underestimate, given extenuating factors like normativity of taste. Beyond this, unless you really believe in your writing, the opportunity cost alone isn’t worth the effort.