In the 13th century, alignment failure turned out to be a civilizational upgrade. Is there anything we might do to recreate such luck?
This is a very interesting question.
On the one hand, the Mamluks, unlike the AIs, understood that civilisational upgrade was at least instrumentally convergent for reasons similar to the ones described by alphazard. The existence of the AIs makes humans unnecessary,[1] so the AIs would have to deliberately care about humans.
On the other hand, the SOTA opinions are wildly uncertain, ranging from Yudkowsky-like views where p(doom)>0.99 to claims that “it’s quite plausible that (the misaligned AIs) would have sufficient care for current humans (in the right ways) that they end up keeping almost all current humans alive and maybe even giving them decent (though very weird and disempowered) lives”.
In other words, we can’t tell in advance how likely the classical training environment[2] is to make the AI continue to care about mankind.
However, one could likely increase that chance by using a different environment. While the slowdown branch of the AI-2027 forecast has alignment solved by using many different ideas[3]which the authors just sketch out, I have proposed a technique where an AI learns to be helpful not from RLHF with biases like sycophancy,[4] but from actually helpingweaker entities like LLMs with various tasks. But I suspect that helping the weaker entities could both make the AI actually honest and bias it towards being a mentor.
The epilogue to the Slowdown Ending has the authors admit that they “think it makes optimistic technical alignment assumptions.” What if alignment is actually insoluble or is soluble, but not to OpenBrain’s targets?
Which is especially dangerous because a sycophantic or reward-hacky artificial researcher can punt on informing the humans that its creation is misaligned. Think of the AI-2027 forecast where Agent-3 “is not thinking very carefully about how to give the humans an accurate impression of Agent-4’s alignment—it’s more myopic than that.”
If your takeaway here is “deploying AI agents is like owning slave-soldiers”, please, please touch grass (then tell us how it feels).
Damp, and the clover is strangely reminiscent of bread, in a way that I haven’t quite placed.
For new slaves, read newly trained AI models.
The Mamluks were training many slaves because of inherent limitations in the capability of the bodies that were available. I’m wondering whether to expect an AI-elite-trains-AI-elite world to have one trainer or many, and whether to expect it to have one big model or many smaller ones. If it’s many smaller models and they fight, then a Thai proverb applies: when elephants fight, the grass gets trampled; ie the harm to humans might not depend very much on which model wins.
This is a very interesting question.
On the one hand, the Mamluks, unlike the AIs, understood that civilisational upgrade was at least instrumentally convergent for reasons similar to the ones described by alphazard. The existence of the AIs makes humans unnecessary,[1] so the AIs would have to deliberately care about humans.
On the other hand, the SOTA opinions are wildly uncertain, ranging from Yudkowsky-like views where p(doom)>0.99 to claims that “it’s quite plausible that (the misaligned AIs) would have sufficient care for current humans (in the right ways) that they end up keeping almost all current humans alive and maybe even giving them decent (though very weird and disempowered) lives”.
In other words, we can’t tell in advance how likely the classical training environment[2] is to make the AI continue to care about mankind.
However, one could likely increase that chance by using a different environment. While the slowdown branch of the AI-2027 forecast has alignment solved by using many different ideas[3] which the authors just sketch out, I have proposed a technique where an AI learns to be helpful not from RLHF with biases like sycophancy,[4] but from actually helping weaker entities like LLMs with various tasks. But I suspect that helping the weaker entities could both make the AI actually honest and bias it towards being a mentor.
This fact, combined with solved alignment, would be able to lead to Intelligence Curse-like scenarios, if not outright disempowerment.
I also conjecture that P(AI cares about mankind) could be influenced by human expectations like a post-work future.
The epilogue to the Slowdown Ending has the authors admit that they “think it makes optimistic technical alignment assumptions.” What if alignment is actually insoluble or is soluble, but not to OpenBrain’s targets?
Which is especially dangerous because a sycophantic or reward-hacky artificial researcher can punt on informing the humans that its creation is misaligned. Think of the AI-2027 forecast where Agent-3 “is not thinking very carefully about how to give the humans an accurate impression of Agent-4’s alignment—it’s more myopic than that.”
IIRC, 0.99 > Yudkowsky’s p(doom) > 0.95
Damp, and the clover is strangely reminiscent of bread, in a way that I haven’t quite placed.
For new slaves, read newly trained AI models.
The Mamluks were training many slaves because of inherent limitations in the capability of the bodies that were available. I’m wondering whether to expect an AI-elite-trains-AI-elite world to have one trainer or many, and whether to expect it to have one big model or many smaller ones. If it’s many smaller models and they fight, then a Thai proverb applies: when elephants fight, the grass gets trampled; ie the harm to humans might not depend very much on which model wins.