I’m interested in why Kaj doesn’t think FAI is a viable solution. Or maybe just agrees with luke that the mainline possibility is failure?
This might be clearer once the survey paper about proposed FAI approaches (as well as other approaches to limiting AI risk) we’re writing becomes public, but suffice to say, IMO nobody so far has managed to propose an FAI approach that wouldn’t be riddled with serious problems. Almost none of them work if we have a hard takeoff, and a soft takeoff might not be any better, due to allowing lots of different AGIs to compete and leading to the kind of evolutionary scenarios like described in the post. If there’s a hard takeoff, you need to devote a lot of time and effort into making the design safe and also be the first one to have your AGI undergo a hard takeoff, two mutually incompatible goals. That’s assuming that you even have a clue of what kind of a design would be safe—something CEV-like could qualify as safe, but currently it remains so vaguely specified that it reads more like a list of applause lights than an actual design, and even getting to the point where we could call it a design feels like it requires solving numerous difficult problems, some of which have remained unsolved for thousands of years, and our remaining time might be counted in tens of years rather than thousands or even hundreds… and so on and so on.
Not saying that it’s impossible, but there are far more failure scenarios than successful ones, and an amazing amount of things would all have to go right in order for us to succeed.
What can be done to improve our chances? I assume more funding for SI is a good idea, and I don’t know how much I can do beyond that (math/philosophy/AI are not my expertise).
We’ll have some suggestions of potentially promising research directions in our survey paper. But if you’re asking about what you yourself can do, then I don’t have anything very insightful to suggest besides the normal recommendations of raising the waterline, spreading the word about these issues, and seeing if there’s any other volunteer work that you could do.
This might be clearer once the survey paper about proposed FAI approaches (as well as other approaches to limiting AI risk) we’re writing becomes public, but suffice to say, IMO nobody so far has managed to propose an FAI approach that wouldn’t be riddled with serious problems. Almost none of them work if we have a hard takeoff, and a soft takeoff might not be any better, due to allowing lots of different AGIs to compete and leading to the kind of evolutionary scenarios like described in the post. If there’s a hard takeoff, you need to devote a lot of time and effort into making the design safe and also be the first one to have your AGI undergo a hard takeoff, two mutually incompatible goals. That’s assuming that you even have a clue of what kind of a design would be safe—something CEV-like could qualify as safe, but currently it remains so vaguely specified that it reads more like a list of applause lights than an actual design, and even getting to the point where we could call it a design feels like it requires solving numerous difficult problems, some of which have remained unsolved for thousands of years, and our remaining time might be counted in tens of years rather than thousands or even hundreds… and so on and so on.
Not saying that it’s impossible, but there are far more failure scenarios than successful ones, and an amazing amount of things would all have to go right in order for us to succeed.
Scary.
What can be done to improve our chances? I assume more funding for SI is a good idea, and I don’t know how much I can do beyond that (math/philosophy/AI are not my expertise).
Waterline stuff is important, too.
We’ll have some suggestions of potentially promising research directions in our survey paper. But if you’re asking about what you yourself can do, then I don’t have anything very insightful to suggest besides the normal recommendations of raising the waterline, spreading the word about these issues, and seeing if there’s any other volunteer work that you could do.