Government control is bad in many ways, but it’s a route to coordination to reduce race dynamics. The government doesn’t take loss-of-control risks seriously, but I think they might once they’re talking to something that’s clearly as smart and agentic as they are.
There are a lot of downsides to this scenario, but if that’s the world we live in, we’d better accept that.
My hope now is that the substantial problems invoked by human-controlled AI/AGI/ASI can be mitigated by wiser AI help; Human-like metacognitive skills will reduce LLM slop because developers have an incentive to make their AI systems more reliable for internal and commercial use, and there are many routes to improving their currently-lacking metacognitve skills.
If everyone in the government got a more-accurate, less-sycophantic answer when they asked Claude 5 and GPT 7 “so what should we be doing with this whole developing AGI thing?” (like “any reasonable aggregation of expert opinion means loss-of-control risks AND human misuse risks are substantial, so you should probably figure out how to be careful; want me to help?”), it might help a lot.
That doesn’t itself solve the problem of government-internal coups, but it might help prevent them if everyone is asking their AIs about the possible routes and defenses.
Government control is bad in many ways, but it’s a route to coordination to reduce race dynamics. The government doesn’t take loss-of-control risks seriously, but I think they might once they’re talking to something that’s clearly as smart and agentic as they are.
There are a lot of downsides to this scenario, but if that’s the world we live in, we’d better accept that.
About a year ago I wrote Whether governments will control AGI is important and neglected. It looks to me now more like the answer is simply yes, and it’s still neglected.
I also think it’s pretty likely that power structures will want AGI aligned to them, which creates different problems of proliferating human-controlled AGI systems More in Instruction-following AGI is easier and more likely than value aligned AGI and If we solve alignment, do we die anyway? from around two years ago.
My hope now is that the substantial problems invoked by human-controlled AI/AGI/ASI can be mitigated by wiser AI help; Human-like metacognitive skills will reduce LLM slop because developers have an incentive to make their AI systems more reliable for internal and commercial use, and there are many routes to improving their currently-lacking metacognitve skills.
If everyone in the government got a more-accurate, less-sycophantic answer when they asked Claude 5 and GPT 7 “so what should we be doing with this whole developing AGI thing?” (like “any reasonable aggregation of expert opinion means loss-of-control risks AND human misuse risks are substantial, so you should probably figure out how to be careful; want me to help?”), it might help a lot.
That doesn’t itself solve the problem of government-internal coups, but it might help prevent them if everyone is asking their AIs about the possible routes and defenses.