The single most important question in AGI safety is: Is the AGI trying to do something that we didn’t intend for it to be trying to do?
Thinking aloud: Is this right?
It seems like “maybe, for a narrow notion of ‘intend’.”
Like, if we build a superintelligence to solve problems that are eluding humans, the superintelligence will be doing things that we didn’t intend (because we didn’t think of them) all the time.
I guess I’d say “the thing we intended for the AGI to be trying to do” can be vague, or described at a meta-level, as opposed to very specific.
I didn’t mean for that sentence to be making a specific controversial claim about alignment targets. I generally see “alignment targets” as an open question (see a footnote in post 10).
Thinking aloud: Is this right?
It seems like “maybe, for a narrow notion of ‘intend’.”
Like, if we build a superintelligence to solve problems that are eluding humans, the superintelligence will be doing things that we didn’t intend (because we didn’t think of them) all the time.
I guess I’d say “the thing we intended for the AGI to be trying to do” can be vague, or described at a meta-level, as opposed to very specific.
I didn’t mean for that sentence to be making a specific controversial claim about alignment targets. I generally see “alignment targets” as an open question (see a footnote in post 10).