Absolutely! It’s helpful to remember we are talking about an intelligence that is comparable to our own. (The great danger only arises with that proximity.) So if you would not feel comfortable with the AI listening in on this conversation (and yes it will do its research, including going back to find this page), you have not understood the problem. The only safety features that will be good enough are those designed with the full knowledge that the AI is sitting at the table with us, having heard every word. That requires a pretty clever answer and clever is where the AI excels!
Furthermore this will be the luxury problem, after humanity has cracked the nut of mutual agreement on our approach to AI. That’s the only way to avoid simply succumbing to infighting; meaning whomever’s first to give the AI what it wants “wins”, (perhaps by being last in line to be sacrificed).
Yes, continued development of AI seems unstoppable. But this brings up another very good point: if humanity cannot become a Singleton in our search for good egalitarian shared values, what is the chance of creating FAI? After years of good work in that direction and perhaps even success in determining a good approximation, what prevents some powerful secret entity like the CIA from hijacking it at the last minute and simply narrowing its objectives for something it determines is a “greater” good?
Our objectives are always better than the other guy’s, and while violence is universally despicable, it is fast, cheap, easy to program and the other guy (including FAI developers) won’t be expecting it. For the guy running the controls, that’s friendly enough. :-)