It seems very difficult to prevent an AI from routing around this limitation by going meta. It can’t actually account for its utility penalty, but can it predict the actions of a similar entity that could, and notice how helpful that would be? An AI that couldn’t predict the behavior of such an entity might be crippled in some serious fundamental ways.
It seems very difficult to prevent an AI from routing around this limitation by going meta. It can’t actually account for its utility penalty, but can it predict the actions of a similar entity that could, and notice how helpful that would be? An AI that couldn’t predict the behavior of such an entity might be crippled in some serious fundamental ways.