Here are some more thoughts on a superintelligence-run persuasion campaign:
Like Daniel wrote in a comment, it’s good to think of Agent-5 as distributed and able to nudge things all over the internet. The nudges could be highly personalized to demographics and individuals, so responsive to the kind of subtle emotional triggers the superintelligence learns about each individual.
It seems many people’s opinions today are already significantly shaped by social media and disinformation. So this makes me think a similar process that’s much more agentic, personalized, and superintelligence-optimized could be very potent.
There’s the possibility of mind-hacking too, though I chose to leave that out of the blogpost.
The CEO is probably well-positioned to take credit for a lot of the benefits Agent-5 seems to bring to the world (some of these benefits are genuine, some illusory).
In an earlier iteration of this scenario I had a military coup rather than this gradual political ascension via persuasion. But then I decided that a superintelligence capable of controlling the robots well enough to disempower the human military would probably also be powerful enough to do something less heavy-handed like what’s in the scenario.
Like Daniel wrote in a comment, it’s good to think of Agent-5 as distributed and able to nudge things all over the internet. The nudges could be highly personalized to demographics and individuals, so responsive to the kind of subtle emotional triggers the superintelligence learns about each individual.
It seems many people’s opinions today are already significantly shaped by social media and disinformation. So this makes me think a similar process that’s much more agentic, personalized, and superintelligence-optimized could be very potent.
Answered to these ideas in my answer to Daniel.
There’s the possibility of mind-hacking too, though I chose to leave that out of the blogpost.
In an earlier iteration of this scenario I had a military coup rather than this gradual political ascension via persuasion. But then I decided that a superintelligence capable of controlling the robots well enough to disempower the human military would probably also be powerful enough to do something less heavy-handed like what’s in the scenario.
I agree that if you have a secret loyalty in wildly superhuman AIs, many things are possible and the success at CEO-takeover seems extremely likely.
The CEO is probably well-positioned to take credit for a lot of the benefits Agent-5 seems to bring to the world (some of these benefits are genuine, some illusory).
I think this is a very different story. If the claim is that even without secret loyalty the CEO would have a 10% chance of taking over, then the secret loyalty part doesn’t seem that important and is distracting imo. A 10% of no-secret-loyalty CEO takeover is a big enough deal on its own?
Like Daniel wrote in a comment, it’s good to think of Agent-5 as distributed and able to nudge things all over the internet
There’s a real missing model or mechanism by which any of the AIs go from “able to nudge large categories of people” to “able to persuade individual actors to do distinct things”. There’s quite a bit of research in the former category but I don’t see there’s any evidence for the latter (Maybe truesight + blackmail would count?)
Here are some more thoughts on a superintelligence-run persuasion campaign:
Like Daniel wrote in a comment, it’s good to think of Agent-5 as distributed and able to nudge things all over the internet. The nudges could be highly personalized to demographics and individuals, so responsive to the kind of subtle emotional triggers the superintelligence learns about each individual.
It seems many people’s opinions today are already significantly shaped by social media and disinformation. So this makes me think a similar process that’s much more agentic, personalized, and superintelligence-optimized could be very potent.
There’s the possibility of mind-hacking too, though I chose to leave that out of the blogpost.
The CEO is probably well-positioned to take credit for a lot of the benefits Agent-5 seems to bring to the world (some of these benefits are genuine, some illusory).
In an earlier iteration of this scenario I had a military coup rather than this gradual political ascension via persuasion. But then I decided that a superintelligence capable of controlling the robots well enough to disempower the human military would probably also be powerful enough to do something less heavy-handed like what’s in the scenario.
Answered to these ideas in my answer to Daniel.
I agree that if you have a secret loyalty in wildly superhuman AIs, many things are possible and the success at CEO-takeover seems extremely likely.
I think this is a very different story. If the claim is that even without secret loyalty the CEO would have a 10% chance of taking over, then the secret loyalty part doesn’t seem that important and is distracting imo. A 10% of no-secret-loyalty CEO takeover is a big enough deal on its own?
There’s a real missing model or mechanism by which any of the AIs go from “able to nudge large categories of people” to “able to persuade individual actors to do distinct things”. There’s quite a bit of research in the former category but I don’t see there’s any evidence for the latter (Maybe truesight + blackmail would count?)