There’s lots of discourse around at the moment about
Will AI go FOOM? With what probability?
Will we die if AI goes FOOM?
Will we die even if AI doesn’t go FOOM?
Does the Halt AI Now case rest on FOOM?
I present a synthesis:
AI might FOOM. If it does, we go from a world much like today’s, straight to dead, with no warning.
If AI doesn’t foom, we go from the AI 2027 scary automation world to dead. Misalignment isn’t solved in slow takeoff worlds.
If you disagree with either of these, you might not want to halt now:
If you think FOOM is impossible, we’ll get plenty of warning to halt later.
If you think slow takeoff is survivable, you might want to press on if the chance of dying in a FOOM is worth the chance of getting to the stars in a slow takeoff world.
To be clear. I think that FOOM is both kinda likely, and that slow takeoff doesn’t save us. I also think that the counter-arguments are pretty weak and strained, and that try pretty much as hard as you can to get a halt or at least be honest that a halt would be good is obviously the best strategy even if you fail and have to rely on some backup strategy.
So the synthesis is:
FOOM isn’t core to the argument that we all die, BUT the possibility of it is a strong motivator for halting sooner rather than waiting later. A sane society would just halt now, obviously, but we don’t have that luxury.
Arguments From Intelligence Explosions (FOOMs)
There’s lots of discourse around at the moment about
Will AI go FOOM? With what probability?
Will we die if AI goes FOOM?
Will we die even if AI doesn’t go FOOM?
Does the Halt AI Now case rest on FOOM?
I present a synthesis:
AI might FOOM. If it does, we go from a world much like today’s, straight to dead, with no warning.
If AI doesn’t foom, we go from the AI 2027 scary automation world to dead. Misalignment isn’t solved in slow takeoff worlds.
If you disagree with either of these, you might not want to halt now:
If you think FOOM is impossible, we’ll get plenty of warning to halt later.
If you think slow takeoff is survivable, you might want to press on if the chance of dying in a FOOM is worth the chance of getting to the stars in a slow takeoff world.
To be clear. I think that FOOM is both kinda likely, and that slow takeoff doesn’t save us. I also think that the counter-arguments are pretty weak and strained, and that try pretty much as hard as you can to get a halt or at least be honest that a halt would be good is obviously the best strategy even if you fail and have to rely on some backup strategy.
So the synthesis is:
FOOM isn’t core to the argument that we all die, BUT the possibility of it is a strong motivator for halting sooner rather than waiting later. A sane society would just halt now, obviously, but we don’t have that luxury.