Xi Jinping seems kind of safety pilled or like he’s going that way as far as I’ve seen. I’ve had around 5-7 people that are goverannce related and knowledgable about the China situation that are datapoints towards that being the case? Maybe 1-2 people against it?
Not on the average likelihood of the deal but it seems to me that china is somewhat safety aligned
I want to update this comment with the following that I saw Dan Fagella share on facebook. I trust Dan’s take here more than I do my own since he was there and so I want to revise the above and say that this seems like a more plausible potential take? (It is still somewhat coherent with the information above just update with the opinion of the people in the middle who often have a better idea of what is going on):
people are like ‘see this reuters article, china is proposing an international ai gov agency!’
after 5 days in shanghai for their world ai conference (waic), i now think china won’t lead on agi safety / the usa has to take the lead on this...
i talked to a bunch of people in the shanghai government and the technical communities. Summary / recs below:
high level review of the event:
- the theme of the event was “global solidarity in the age of ai”
- major western speakers included stuart russell, helen toner, dan hendrycks, and others
- chinese companies showcased ai tools and platforms
- governance and safety were mentioned often in panels
- much talk of transparency, controllability, agent interactions
off-mic conversations told a different story;
- multiple chinese researchers and local officials said: “we’re only talking about ai safety because trump isn’t”
- it’s posturing, not principle. its knowing that trump is not focused on safety, and it leaves a gap for china to look virtuous and to jockey for a perceived position of tech leadership
- it is reactive to the USA, not proactive in terms of prioritizing agi safety for its own sake (geopolitical optics move, not a values shift)
technical culture in china:
- in sf or boston, you can find meetups on alignment
- you can talk about failure modes, governance, strategy
- in china, those conversations don’t really exist
- there’s no “internal discourse” about alignment or agi risk at leading labs like deepseek, they don’t talk about it publicly and they don’t seem to discuss it much in-house
- people want to beat the benchmarks (i.e. “show me what makes the models look safe so i can score high”), not reflect on long-term risks
- its top-down “make the boss happy” / “don’t rock the boat” when it comes to lab employees (discussing agi’s dangers is rocking the boat—not a good thing to do in chinese culture generally)
- no philanthropic funding, no grassroots agi safety orgs (no equivalents of farAI, METR, redwood research, ARIA, etc)
what surprised me:
- i assumed china’s leadership could act top-down to prevent agi loss of control (because Xi is very smart and understands ai’s importance, and because china is very sensative to anything that would shake up the ruling party’s grip on power)
- but it seems xi’s government isn’t philosophically moved by runaway ai, many people i spoke with think that Xi and his top leaders won’t really act on agi safety until the usa does
- there is remarkably little by way of discussion or think tanks / non-profits around agi safety in china, outlandishly less than the anglosphere
what i think needs to happen if international agi coordination is ever going to happen:
- even if the chinese effort to have “solidarity in the ai age” is posturing, it creates an opening for track 2 dialogue with china, and we should do as much of that (among technical leaders, safety thinkers, academics, etc) as we can
2- we need WAY, WAY more funding going to starting agi safety orgs / events / discourse in china—the few peopel who care cannot find funding for ‘safety’ like people do in the usa / uk
3- the usa needs to wake up its citizens to care about agi risk, otherwise the president / mainstream media will never care. theory of change:
- the bottom of our power pyramid (citizens) needs to flip in order for the top of our pyramid (president) to ACT on agi risk
- only when the top of america’s pyramid (president) acts on agi risk with china seriously act on agi risk
- only then to we have a serious window for coordination
that is all. if you know more about china’s agi safety / technical ecosystem that adds luster to this picture let me know
https://www.lesswrong.com/posts/PfyTyZKASezFynYmv/?commentId=ay33KErjBsCLDqfZa
Xi Jinping seems kind of safety pilled or like he’s going that way as far as I’ve seen. I’ve had around 5-7 people that are goverannce related and knowledgable about the China situation that are datapoints towards that being the case? Maybe 1-2 people against it?
Not on the average likelihood of the deal but it seems to me that china is somewhat safety aligned
I want to update this comment with the following that I saw Dan Fagella share on facebook. I trust Dan’s take here more than I do my own since he was there and so I want to revise the above and say that this seems like a more plausible potential take? (It is still somewhat coherent with the information above just update with the opinion of the people in the middle who often have a better idea of what is going on):
China/the CCP have had to actually try to align social media algorithms, for better/worse and several members have a phd in computer science.
I think they’re generally better informed on this than every government I’ve seen