But the AI police have no such axiomatic limitation on restricting our freedom. They are empowered to restrict our freedom, if they think we are restricting the freedom of others. So if they reason that human communication restricts the freedom of others, because providing information sometimes reduces a person’s range of likely actions, they will stop anyone from talking to anyone else.
That’s one type of “bug in the system”—where you might even get the “values” or “laws” right, but they are implemented wrongly. Another problem is if you manage to forget something important—what if there was a fifth law that’s needed for human happiness? Would the system allow you to add it later? This is why people talk about figuring out what humans really want, or really want to want, by studying neuroscience, rather than just by private reasoning, and then using that as the basis for utopia.
Despite all that, I think it’s appropriate to have some informal discussion of “how utopia would work”. But that can only produce a first draft which then needs to be judged by much more sophisticated criteria. The first draft might turn out to be 90% right, but it also might turn out to be 90% wrong.
But the AI police have no such axiomatic limitation on restricting our freedom. They are empowered to restrict our freedom, if they think we are restricting the freedom of others. So if they reason that human communication restricts the freedom of others, because providing information sometimes reduces a person’s range of likely actions, they will stop anyone from talking to anyone else.
That’s one type of “bug in the system”—where you might even get the “values” or “laws” right, but they are implemented wrongly. Another problem is if you manage to forget something important—what if there was a fifth law that’s needed for human happiness? Would the system allow you to add it later? This is why people talk about figuring out what humans really want, or really want to want, by studying neuroscience, rather than just by private reasoning, and then using that as the basis for utopia.
Despite all that, I think it’s appropriate to have some informal discussion of “how utopia would work”. But that can only produce a first draft which then needs to be judged by much more sophisticated criteria. The first draft might turn out to be 90% right, but it also might turn out to be 90% wrong.