I think “Luck could be enough” should be the strong default on priors,2 so in some sense I don’t think I owe tons of argumentation here (I think the burden is on the other side).
I agree with this being the default and the burden being on the other side. At the same time, I don’t think of it as a strong default.
Here’s a frame that I have that already gets me to a more pessimistic (updated) prior:
It has almost never happened that people who developed and introduced a revolutionary new technology displayed a lot of foresight about its long-term consequences. For instance, there were comparatively few efforts at major social media companies to address ways in which social media might change society for the worse. The same goes for the food industry and the obesity epidemic or online dating and its effects on single parenthood rates. When people invent cool new technology, it makes the world better on some metrics but creates new problems on its own. The whole thing is accelerating and feels out of control.
It feels out of control because even if we get cool new things from tech progress, we don’t seem to be getting any better at fixing the messiness that comes with it (misaligned incentives/goodhearting, other Molochian forces, world-destroying tech becoming ever more accessible). Your post says “a [] story of avoiding catastrophe by luck.” This framing makes it sound like things would be fine by default if it isn’t for some catastrophe happening. However, humans have never seemed particularly “in control” over technological progress. For things to go well, we need the opposite of a catastrophe – a radical change towards the upside. We have to solve massive coordination problems and hope for a technology that gives us god-like power, finally putting sane and compassionate forces in control over the future. It so happens that we can tell a coherent story about how AI might do this for us. But to say that it might go right just by luck – I don’t know, that seems far-fetched!
All of that said, I don’t think we can get very far arguing from priors. What carries by far the most weight are arguments about alignment difficulty, takeoff speeds, etc. And I think it’s a reasonable view to say that it’s very unlikely that any researchers currently know enough to make highly confident statements about these variables. (Edit: So, I’m not sure we disagree too much – I think I’m more pessimistic about the future than you are, but I’m probably not as pessimistic as the position you’re arguing against in this post. I mostly wanted to make the point that I think the “right” priors support at least moderate pessimism, which is a perspective I find oddly rare among EAs.)
FWIW, it’s not obvious to me that slow takeoff is best. Fast takeoff at least gives you god-like abilities early on, which are useful from a perspective of “we were never particularly in control over history; lots of underlying problems need fixing before we pass a point of no return.” By contrast, with slow takeoff, coordination problems seem more difficult because (at least by default) there will be more actors using AIs in some ways or other and it’s not obvious that the AIs in a slow-takeoff scenario will be all that helpful at facilitating coordination.
I agree with this being the default and the burden being on the other side. At the same time, I don’t think of it as a strong default.
Here’s a frame that I have that already gets me to a more pessimistic (updated) prior:
It has almost never happened that people who developed and introduced a revolutionary new technology displayed a lot of foresight about its long-term consequences. For instance, there were comparatively few efforts at major social media companies to address ways in which social media might change society for the worse. The same goes for the food industry and the obesity epidemic or online dating and its effects on single parenthood rates. When people invent cool new technology, it makes the world better on some metrics but creates new problems on its own. The whole thing is accelerating and feels out of control.
It feels out of control because even if we get cool new things from tech progress, we don’t seem to be getting any better at fixing the messiness that comes with it (misaligned incentives/goodhearting, other Molochian forces, world-destroying tech becoming ever more accessible). Your post says “a [] story of avoiding catastrophe by luck.” This framing makes it sound like things would be fine by default if it isn’t for some catastrophe happening. However, humans have never seemed particularly “in control” over technological progress. For things to go well, we need the opposite of a catastrophe – a radical change towards the upside. We have to solve massive coordination problems and hope for a technology that gives us god-like power, finally putting sane and compassionate forces in control over the future. It so happens that we can tell a coherent story about how AI might do this for us. But to say that it might go right just by luck – I don’t know, that seems far-fetched!
All of that said, I don’t think we can get very far arguing from priors. What carries by far the most weight are arguments about alignment difficulty, takeoff speeds, etc. And I think it’s a reasonable view to say that it’s very unlikely that any researchers currently know enough to make highly confident statements about these variables. (Edit: So, I’m not sure we disagree too much – I think I’m more pessimistic about the future than you are, but I’m probably not as pessimistic as the position you’re arguing against in this post. I mostly wanted to make the point that I think the “right” priors support at least moderate pessimism, which is a perspective I find oddly rare among EAs.)
FWIW, it’s not obvious to me that slow takeoff is best. Fast takeoff at least gives you god-like abilities early on, which are useful from a perspective of “we were never particularly in control over history; lots of underlying problems need fixing before we pass a point of no return.” By contrast, with slow takeoff, coordination problems seem more difficult because (at least by default) there will be more actors using AIs in some ways or other and it’s not obvious that the AIs in a slow-takeoff scenario will be all that helpful at facilitating coordination.