FWIW I don’t think Agent-5 needs to be vastly superhuman at politics to succeed in this scenario, merely top-human level. Analogy: A single humanoid robot might need to be vastly superhuman at fighting to take out the entire US army in a land battle. But a million humanoid robots could probably do it if they were merely expert at fighting. Agent-5 isn’t a single agent, it’s a collective of millions.
I think this seems more dicey for the AI if it’s not vastly superhuman at politics. There are forces pushing for many different people to be in power quite hard, so I think it takes somewhat forceful actions to move probabilities by a Bayes factor above 10, which makes me skeptical that something like the subtle actions in the essay would have the intended effect.
I would guess the AI’s subtle policy actions would not be much more coordinated and high impact position than what TikTok could have done a few years ago, and I don’t think TikTok was ever in a position where it could have put a much more pro-CCP candidate in power if it wanted to (if the pro-CCP candidate started from a similar position of strength as the CEO) - my guess is that it’s hard to massively shift the balance of power without it being obvious to the people losing that something nefarious is going on.
My guess is that the AI could try to have a massive cover-up operation to discredit people who criticize the somewhat obvious media manipulation—but then the thing the AI strategy is more like “somewhat blatantly do bad things and then do a giant cover-up” rather than “take actions so subtle nobody would ever figure it out even if they had access to all the logs”.
In my scenario I was imagining Agent-5 to be superhuman at politics. But I tentatively think [just top-human at politics + being extremely well-informed about what every human actor wants and knows + being able to have a coordinated influence on all these human actors] already has a big shot at leading to the story’s events.
It’s also important that the CEO starts from a relative position of power and prestige, and is not some random guy. He’s the kind of person who could rise to power the way he does (at least at first) if he was lucky and had political acumen. Maybe I should have emphasized this point more in the scenario.
But a million humanoid robots could probably do it if they were merely expert at fighting. Agent-5 isn’t a single agent, it’s a collective of millions.
Something close to this might be a big reason why Amdahl’s law/parallelization bottlenecks on the software singularity might not matter, because the millions of AIs are much, much closer to one single AI doing deep serial research than it is to an entire human field with thousands or millions of people.
There is something something lack of powerful in context learning, where currently millions of AI are bsaically one AI because they can’t change rapidly in response to new information, but once they can they will be a tree of AI from copying the ones with insights.
FWIW I don’t think Agent-5 needs to be vastly superhuman at politics to succeed in this scenario, merely top-human level. Analogy: A single humanoid robot might need to be vastly superhuman at fighting to take out the entire US army in a land battle. But a million humanoid robots could probably do it if they were merely expert at fighting. Agent-5 isn’t a single agent, it’s a collective of millions.
I think this seems more dicey for the AI if it’s not vastly superhuman at politics. There are forces pushing for many different people to be in power quite hard, so I think it takes somewhat forceful actions to move probabilities by a Bayes factor above 10, which makes me skeptical that something like the subtle actions in the essay would have the intended effect.
I would guess the AI’s subtle policy actions would not be much more coordinated and high impact position than what TikTok could have done a few years ago, and I don’t think TikTok was ever in a position where it could have put a much more pro-CCP candidate in power if it wanted to (if the pro-CCP candidate started from a similar position of strength as the CEO) - my guess is that it’s hard to massively shift the balance of power without it being obvious to the people losing that something nefarious is going on.
My guess is that the AI could try to have a massive cover-up operation to discredit people who criticize the somewhat obvious media manipulation—but then the thing the AI strategy is more like “somewhat blatantly do bad things and then do a giant cover-up” rather than “take actions so subtle nobody would ever figure it out even if they had access to all the logs”.
In my scenario I was imagining Agent-5 to be superhuman at politics. But I tentatively think [just top-human at politics + being extremely well-informed about what every human actor wants and knows + being able to have a coordinated influence on all these human actors] already has a big shot at leading to the story’s events.
It’s also important that the CEO starts from a relative position of power and prestige, and is not some random guy. He’s the kind of person who could rise to power the way he does (at least at first) if he was lucky and had political acumen. Maybe I should have emphasized this point more in the scenario.
Something close to this might be a big reason why Amdahl’s law/parallelization bottlenecks on the software singularity might not matter, because the millions of AIs are much, much closer to one single AI doing deep serial research than it is to an entire human field with thousands or millions of people.
There is something something lack of powerful in context learning, where currently millions of AI are bsaically one AI because they can’t change rapidly in response to new information, but once they can they will be a tree of AI from copying the ones with insights.