The non-generic part is “turning an AI on”, and “good idea” is an epistemic consideration on part of the designers, not reference to actual outcome in a reality.
I heard that sentence attributed to Yudkowsky on some podcasts, and it makes sense as an umbrella desideratum after pivoting from shared-extrapolated-values AI to pivotal act AI (as described on arbital), since with that goal there doesn’t appear to be a more specific short summary anymore. In context of that sentence as I mentioned it there is discussion of humans with augmented intelligence, so pivotal act (specialized tool) AI is more centrally back on the table (even as it still seems prudent to plan for long AGI timelines in our world as it is, to avoid abandoning that possibility only to arrive at it unprepared).
The non-generic part is “turning an AI on”, and “good idea” is an epistemic consideration on part of the designers, not reference to actual outcome in a reality.
I heard that sentence attributed to Yudkowsky on some podcasts, and it makes sense as an umbrella desideratum after pivoting from shared-extrapolated-values AI to pivotal act AI (as described on arbital), since with that goal there doesn’t appear to be a more specific short summary anymore. In context of that sentence as I mentioned it there is discussion of humans with augmented intelligence, so pivotal act (specialized tool) AI is more centrally back on the table (even as it still seems prudent to plan for long AGI timelines in our world as it is, to avoid abandoning that possibility only to arrive at it unprepared).