It’s not possible to explain what you don’t know, to answer a question you can’t state, and “intelligence” doesn’t save from this trouble, doesn’t open the floodgates to arbitrary helpfulness, resolving any difficulties you have. It just does its thing really well, but it’s up to its designers to choose the right thing as its optimization criterion. Doing the wrong thing very well, on the other hand, is in no one’s interest. This is a brittle situation, where vagueness in understanding the goal leads to arbitrary and morally desolate outcomes.
It’s not possible to explain what you don’t know, to answer a question you can’t state, and “intelligence” doesn’t save from this trouble, doesn’t open the floodgates to arbitrary helpfulness, resolving any difficulties you have. It just does its thing really well, but it’s up to its designers to choose the right thing as its optimization criterion. Doing the wrong thing very well, on the other hand, is in no one’s interest. This is a brittle situation, where vagueness in understanding the goal leads to arbitrary and morally desolate outcomes.