It’s a good start, but I don’t think this is a reasonably exhaustive list, since I don’t find myself on it :)
My position is closest to your number 3: “ASI will not want to take over or destroy the world.” Mostly because “want” is a very anthropomorphic concept. The Orthogonality Thesis is not false, but inapplicable, since AI are so different from humans. They did not evolve to survive, they were designed to answer questions.
It will be possible to coordinate to prevent any AI from being given deliberately dangerous instructions, and also any unintended consequences will not be that much of a problem
I do not think it will be possible, and I expect some serious calamities from people intentionally or accidentally giving an AI “deliberately dangerous instructions”. I just wouldn’t expect it to result in systematic extermination of all life on earth, since the AI itself does not care in the same way humans do. Sure, it’s a dangerous tool to wield, but it is not a malevolent one. Sort of 3-b-iv, but not quite.
But mostly the issue with doomerism I see is that the Knightian uncertainty on any non-trivial time frame: there will be black swans in all directions, just like there have been lately (for example, no one expected near-human-level LARPing that LLMs do, while not being in any way close to a sentient agent).
To be clear, I expect the world to change quickly and maybe even unrecognizably in the next decade or two, with lots of catastrophic calamities, but the odds of complete “destruction of all value”, the way Zvi puts it, cannot be evaluated at this point with any confidence. The only way to get this confidence is to walk the walk. Pausing and being careful and deliberate about each step does not seem to make sense, at least not yet.
I see that as being related to current AIs not being particularly agentic. I agree in the short run, but in the long run there’s a lot of pressure to make AIs more agentic and some of those dangerous instructions will be pointed in the direction of increased agency too.
It’s a good start, but I don’t think this is a reasonably exhaustive list, since I don’t find myself on it :)
My position is closest to your number 3: “ASI will not want to take over or destroy the world.” Mostly because “want” is a very anthropomorphic concept. The Orthogonality Thesis is not false, but inapplicable, since AI are so different from humans. They did not evolve to survive, they were designed to answer questions.
I do not think it will be possible, and I expect some serious calamities from people intentionally or accidentally giving an AI “deliberately dangerous instructions”. I just wouldn’t expect it to result in systematic extermination of all life on earth, since the AI itself does not care in the same way humans do. Sure, it’s a dangerous tool to wield, but it is not a malevolent one. Sort of 3-b-iv, but not quite.
But mostly the issue with doomerism I see is that the Knightian uncertainty on any non-trivial time frame: there will be black swans in all directions, just like there have been lately (for example, no one expected near-human-level LARPing that LLMs do, while not being in any way close to a sentient agent).
To be clear, I expect the world to change quickly and maybe even unrecognizably in the next decade or two, with lots of catastrophic calamities, but the odds of complete “destruction of all value”, the way Zvi puts it, cannot be evaluated at this point with any confidence. The only way to get this confidence is to walk the walk. Pausing and being careful and deliberate about each step does not seem to make sense, at least not yet.
I see that as being related to current AIs not being particularly agentic. I agree in the short run, but in the long run there’s a lot of pressure to make AIs more agentic and some of those dangerous instructions will be pointed in the direction of increased agency too.