AGI will require both learning and planning, the latter of which is already then a learned mesa optimizer. And AGI may help create new AGI, which is also a form of mesa-optimization. Yes it’s unavoidable.
To create friendly but powerful AGI, we need to actually align it to human values. Creating friendly but weak AI doesn’t matter.
AGI will require both learning and planning, the latter of which is already then a learned mesa optimizer. And AGI may help create new AGI, which is also a form of mesa-optimization. Yes it’s unavoidable.
To create friendly but powerful AGI, we need to actually align it to human values. Creating friendly but weak AI doesn’t matter.