Oh, now I understand the moral dilemma. Something like an Ineffective Friendly AI, which uses sqrt(x) or even log(x) resources for doing actually Friendly things, and the rest of them are wasted on doing something that is not really harmful, just completely useless; with no perspective to ever become more effective.
Would you turn that off? And perhaps risk that the next AI will turn out not to be Friendly, or it will be Friendly, but even more wasteful than the old one, however better at defending itself. Or would you let it run and accept that the price is turning most of the universe into bullshitronium?
I guess for a story it is a good thing when both sides can be morally defended.
Thanks. Yes, I was thinking of an AI that is both superintelligent and technically Friendly, but about log(x)^10 of the benefit from the intelligence explosion is actually received by humans. The AI just sets up its own cult and meditates for most of the day, thinking of how to wring more money out of its adoring fans. Are there ways to set up theoretical frameworks that avoid scenarios vaguely similar to that? If so, how?
Oh, now I understand the moral dilemma. Something like an Ineffective Friendly AI, which uses sqrt(x) or even log(x) resources for doing actually Friendly things, and the rest of them are wasted on doing something that is not really harmful, just completely useless; with no perspective to ever become more effective.
Would you turn that off? And perhaps risk that the next AI will turn out not to be Friendly, or it will be Friendly, but even more wasteful than the old one, however better at defending itself. Or would you let it run and accept that the price is turning most of the universe into bullshitronium?
I guess for a story it is a good thing when both sides can be morally defended.
Thanks. Yes, I was thinking of an AI that is both superintelligent and technically Friendly, but about log(x)^10 of the benefit from the intelligence explosion is actually received by humans. The AI just sets up its own cult and meditates for most of the day, thinking of how to wring more money out of its adoring fans. Are there ways to set up theoretical frameworks that avoid scenarios vaguely similar to that? If so, how?