I’m confused here Matthew. It seems to me that it is highly probable that AI systems which want takeover vs ones that want moderate power combined with peaceful coexistence with humanity… are pretty hard to distinguish early on. And early on is when it’s most important for humanity to distinguish between them, before those systems have gotten power and thus we can still stop them.
Picture a merciless un-aging sociopath capable of duplicating itself easily and rapidly were on a trajectory of gaining economic, political, and military power with the aim of acquiring as much power as possible. Imagine that this entity has the option of making empty promises and highly persuasive lies to humans in order to gain power, with no intention of fulfilling any of those promises once it achieves enough power.
That seems like a scary possibility to me. And I don’t know how I’d trust an agent which seemed like it could be this, but was making really nice sounding promises. Even if it was honoring its short-term promises while still under the constraints of coercive power from currently dominant human institutions, I still wouldn’t trust that it would continue keeping its promises once it had the dominant power.
I’m confused here Matthew. It seems to me that it is highly probable that AI systems which want takeover vs ones that want moderate power combined with peaceful coexistence with humanity… are pretty hard to distinguish early on. And early on is when it’s most important for humanity to distinguish between them, before those systems have gotten power and thus we can still stop them.
Picture a merciless un-aging sociopath capable of duplicating itself easily and rapidly were on a trajectory of gaining economic, political, and military power with the aim of acquiring as much power as possible. Imagine that this entity has the option of making empty promises and highly persuasive lies to humans in order to gain power, with no intention of fulfilling any of those promises once it achieves enough power.
That seems like a scary possibility to me. And I don’t know how I’d trust an agent which seemed like it could be this, but was making really nice sounding promises. Even if it was honoring its short-term promises while still under the constraints of coercive power from currently dominant human institutions, I still wouldn’t trust that it would continue keeping its promises once it had the dominant power.