It is a perfectly good question though. And yes, the “AI safety” people have heard it before, but their answer, insofar as they deign to provide one, is still shit.
The response that comes closest to an answer seems to be “imagine this thing we call ‘humanity’s CEV.’ It would be nice if such a thing existed, so we’re going to assume it does.”
That’s not the only or even most recent answer. discussion of CEV dates back to around 2004 up to around 2010–2013 or so, and mostly predates Value Learning, which dates back to about 2011. That’s more “Imagine this thing we call ‘human values’ that’s about what humans want. It would be helpful if such a thing existed and could be systematized, and it clearly kind of does, since we’ve been studying it for over 3000 years, and subfields of it take up nearly half the Dewey Decimal system: pretty much every soft science, art, and craft. So we’re going to have AI continue that research with us.”
It is a perfectly good question though. And yes, the “AI safety” people have heard it before, but their answer, insofar as they deign to provide one, is still shit.
The response that comes closest to an answer seems to be “imagine this thing we call ‘humanity’s CEV.’ It would be nice if such a thing existed, so we’re going to assume it does.”
That’s not the only or even most recent answer. discussion of CEV dates back to around 2004 up to around 2010–2013 or so, and mostly predates Value Learning, which dates back to about 2011. That’s more “Imagine this thing we call ‘human values’ that’s about what humans want. It would be helpful if such a thing existed and could be systematized, and it clearly kind of does, since we’ve been studying it for over 3000 years, and subfields of it take up nearly half the Dewey Decimal system: pretty much every soft science, art, and craft. So we’re going to have AI continue that research with us.”