would a true Oracle AI that didn’t understand that humans could get bored really be that bad?
The important difficulty is not in making an AGI realize that humans can get bored, it is in making it realize that this is undesirable. Also, it would have to be able to figure out whether it is desirable to remove humans’ boredom, effectively wireheading them in order to make them willing to enjoy repetitive activities.
I do agree that this could be less of a problem for an Oracle non-AI than for a goal-directed system, depending on other details of the design.
The important difficulty is not in making an AGI realize that humans can get bored, it is in making it realize that this is undesirable. Also, it would have to be able to figure out whether it is desirable to remove humans’ boredom, effectively wireheading them in order to make them willing to enjoy repetitive activities.
I do agree that this could be less of a problem for an Oracle non-AI than for a goal-directed system, depending on other details of the design.