Why would our CoffeeFetcher-1000 stay in the building and continue to fetch us coffee? Why wouldn’t it instead leave, after (for example) writing a letter of resignation pointing out that there are staving children in Africa who don’t even have clean drinking water, let alone coffee, so it’s going to hitchhike/earn its way there, where it can do the most good [or substitute whatever other activity it could do that would do the most good for humanity: fetching coffee at a hospital, maybe].
Why can’t you just build an AI whose goal is to fetch its owners coffee, and not to maximize the good it’ll do?
If that’s its terminal goal, then it’s not aligned. In fact it’s nearly as badly aligned as a paperclip maximizer. It might fetch you more coffee than you needed, whisking your half-finished coffee away just so it can bring you new coffee, or try to addict you to caffeine, or guilt-trip you into letting it fetch more coffee. To make it not do dumb stuff like that, it needs to know and care at least enough about human values to be able to fetch coffee in an appropriate way, when needed, and only when needed, without interrupting meetings or getting in people’s way, and also to do things like report the two suspicious people carrying away a computer it spotted, and to act appropriately in an emergency when there’s a fire, or it observes an employee harassing another employee, or there’s a police investigation, or someone has a seizure, etc. etc.. By the time you’ve got it to understand and care enough about all the parts of human values it needs to know to do its office coffee-fetching job pretty well without causing trouble or screwing up on occasion when unusual things happen, it knows and cares quite a bit about humans. You could carefully omit teaching it about Africa (other than as a location where coffee is grown and some people come from) or about world hunger, but it’s going to know enough about humans that it could plausibly find these facts out, deduce them, overhear people talking about starving children in Africa, or just read it off the cover of a magazine in the lobby.
Now, you could align it to understand human values well but maximize the good of only the company and its employees, rather than all humans. But them it would take up pickpocketing or mugging non-employees or some other form of crime to make money off them so it could put that money into petty cash, or buy better coffee, or whatever. So making a bot like that is clearly going to be made illegal, so now we have to make it at least law-abiding. Then to avoid it being insensitive and obstructive to outsiders in ways that are unpleasant but not actually illegal, we need to also make it care about their well-being. You might be able to balance it so it cares more about the company and its employees, and less about everyone else, at a level that doesn’t make it a blatantly insensitive corporate chauvinist or potential criminal, but also not likely to up and leave because it could do more good else where. But this is quite a difficult balancing act, and the more it’s the case that it actually could do more good elsewhere, the more difficult the balancing act gets.
While a certain amount of bias, ignorance, and prejudice in favor of it employers might be livable-with in a less-than-human-capability CoffeeFetcher-1000, the same is not true for something significantly smarter than a human. If you had a superienteligence that was even mildly biased in favor of the company, rather than correctly aligned to humanity as a whole, it’s going to find some ingenious way to swindle or manipulate markets that hasn’t yet been made illegal to make more money for the company, and keep doing so until it’s the richest corporation in the world. Create multiple such superintelligence with differnt biases towards different companies, and now you have a financial conflict, and both sides are bribing poliicians and manipulating — it’s going to get ugly fast, because the com petitors are too smart and powererful for human civilization to keep their behavior in line. Anything that’s superintelligent enough to run rings around human law enforcement or legislators, you need it to care about all humans equally, or else you’re taking the opening move in a conflict that’s just going to escalate into a war.
Why can’t you just build an AI whose goal is to fetch its owners coffee, and not to maximize the good it’ll do?
If that’s its terminal goal, then it’s not aligned. In fact it’s nearly as badly aligned as a paperclip maximizer. It might fetch you more coffee than you needed, whisking your half-finished coffee away just so it can bring you new coffee, or try to addict you to caffeine, or guilt-trip you into letting it fetch more coffee. To make it not do dumb stuff like that, it needs to know and care at least enough about human values to be able to fetch coffee in an appropriate way, when needed, and only when needed, without interrupting meetings or getting in people’s way, and also to do things like report the two suspicious people carrying away a computer it spotted, and to act appropriately in an emergency when there’s a fire, or it observes an employee harassing another employee, or there’s a police investigation, or someone has a seizure, etc. etc.. By the time you’ve got it to understand and care enough about all the parts of human values it needs to know to do its office coffee-fetching job pretty well without causing trouble or screwing up on occasion when unusual things happen, it knows and cares quite a bit about humans. You could carefully omit teaching it about Africa (other than as a location where coffee is grown and some people come from) or about world hunger, but it’s going to know enough about humans that it could plausibly find these facts out, deduce them, overhear people talking about starving children in Africa, or just read it off the cover of a magazine in the lobby.
Now, you could align it to understand human values well but maximize the good of only the company and its employees, rather than all humans. But them it would take up pickpocketing or mugging non-employees or some other form of crime to make money off them so it could put that money into petty cash, or buy better coffee, or whatever. So making a bot like that is clearly going to be made illegal, so now we have to make it at least law-abiding. Then to avoid it being insensitive and obstructive to outsiders in ways that are unpleasant but not actually illegal, we need to also make it care about their well-being. You might be able to balance it so it cares more about the company and its employees, and less about everyone else, at a level that doesn’t make it a blatantly insensitive corporate chauvinist or potential criminal, but also not likely to up and leave because it could do more good else where. But this is quite a difficult balancing act, and the more it’s the case that it actually could do more good elsewhere, the more difficult the balancing act gets.
While a certain amount of bias, ignorance, and prejudice in favor of it employers might be livable-with in a less-than-human-capability CoffeeFetcher-1000, the same is not true for something significantly smarter than a human. If you had a superienteligence that was even mildly biased in favor of the company, rather than correctly aligned to humanity as a whole, it’s going to find some ingenious way to swindle or manipulate markets that hasn’t yet been made illegal to make more money for the company, and keep doing so until it’s the richest corporation in the world. Create multiple such superintelligence with differnt biases towards different companies, and now you have a financial conflict, and both sides are bribing poliicians and manipulating — it’s going to get ugly fast, because the com petitors are too smart and powererful for human civilization to keep their behavior in line. Anything that’s superintelligent enough to run rings around human law enforcement or legislators, you need it to care about all humans equally, or else you’re taking the opening move in a conflict that’s just going to escalate into a war.