The focus on categorizing negative outcomes seems to obscure a lot of relevant detail. If you broaden the view to include things from non-takeover to peaceful cooperation, it would probably be evident that the boundaries are soft and scenarios near the edge aren’t as definitely bad as this makes them appear. I think we might learn more about possible coexistence and how thing might turn out well if we spend less time focused on imagined disasters.
First I’d point out that scenarios like Drexler’s “Reframing Superintelligence” aren’t that far from the *Flash Economy* scenario, and there’s probably a gradual blend across the boundary. My current thinking is similar to that, though I characterize it as AIs participating in an economy, which includes competition and cooperation. You get ahead in an economy by providing services that others are willing to pay for. If you do that for very long, you are incentivized to learn that cooperation (filling a need) is the way to garner resources that give you the ability to get more done.
Second I’d point out that there are many voices currently arguing that the largest companies of today (FANG, often) have too much power and are monopolizing societies’ resources. I’d just like to contrast that with the same arguments made in earlier eras about IBM, Microsoft, AT&T, Standard Oil, big Hollywood, the railroad trusts, the A&P stores, and many others. If you have the right viewpoint, you can look at an economy where someone is earning more money than you think is just, and believe that they’re going to own the world in another year if nothing is done to stop them. It’s usually competition and innovation that keeps this from happening and not government or other concerted action.
The focus on categorizing negative outcomes seems to obscure a lot of relevant detail. If you broaden the view to include things from non-takeover to peaceful cooperation, it would probably be evident that the boundaries are soft and scenarios near the edge aren’t as definitely bad as this makes them appear. I think we might learn more about possible coexistence and how thing might turn out well if we spend less time focused on imagined disasters.
I think this is probably correct to a degree—we do say we’re looking at AI takeover scenarios, and all but one of our scenarios (WFLL 1) eventually result in some terrible outcome for humanity and human values, like extinction. However, we do briefly discuss the possibility that the ‘takeover’ might be ambiguously bad—in WFLL 1 we said there was a 50:50 chance of actual human extinction, and in that scenario it’s even possible that something like modern human civilisation would continue. However, the fact that there’s some blurring at the edges does not mean that the possible scenarios fill up a continuous spectrum of badness. I think it’s quite likely that the possible outcomes are pretty bimodal, even if it’s not literally a binary of ‘we go extinct’ vs ‘techno-utopia’, we still either keep control of the future or lose more and more of it as time goes on. If we’re in a future where we’re cooperating and competing with TAIs in such a way that we can get what we want and influence the future, that’s not a takeover scenario.
Second I’d point out that there are many voices currently arguing that the largest companies of today (FANG, often) have too much power and are monopolizing societies’ resources. I’d just like to contrast that with the same arguments made in earlier eras about IBM, Microsoft, AT&T, Standard Oil, big Hollywood, the railroad trusts, the A&P stores, and many others.
I don’t disagree with this point and while there are differences of opinion my own view is that big tech isn’t a uniquely bad monopoly, i.e. not worse than things like Hollywood, Standard Oil etc. However, in this case, we have a unique reason to think that these problems will just keep getting worse—namely the presence of nonhuman actors that are power-seeking. So maybe the analogy that it will be like ‘big tech but worse’ doesn’t quite fit because there’s a specific reason as to why things would get much worse than any historical example of a monopoly.
AIs participating in an economy, which includes competition and cooperation. You get ahead in an economy by providing services that others are willing to pay for. If you do that for very long, you are incentivized to learn that cooperation (filling a need) is the way to garner resources that give you the ability to get more done.
This is one of the things we discuss in terms of alignment being ‘hackable’ - our term for how easy/hard it is to keep systems behaving well with changes to their incentives and incremental fixes. If alignment is quite hackable (i.e. there aren’t deep technical reasons systems will go wrong in ways that won’t be detected), then what you’re describing would work. If systems do the wrong thing we punish them by denying them the resources/rewards they’re seeking. AAFS/Production Web describe this strategy working for a while and then abruptly failing because of hidden vulnerabilities, because this post is about AI takeover stories, but it’s definitely possible that it doesn’t fail, and we end up in a multipolar outcome that’s persistently good.
The focus on categorizing negative outcomes seems to obscure a lot of relevant detail. If you broaden the view to include things from non-takeover to peaceful cooperation, it would probably be evident that the boundaries are soft and scenarios near the edge aren’t as definitely bad as this makes them appear. I think we might learn more about possible coexistence and how thing might turn out well if we spend less time focused on imagined disasters.
First I’d point out that scenarios like Drexler’s “Reframing Superintelligence” aren’t that far from the *Flash Economy* scenario, and there’s probably a gradual blend across the boundary. My current thinking is similar to that, though I characterize it as AIs participating in an economy, which includes competition and cooperation. You get ahead in an economy by providing services that others are willing to pay for. If you do that for very long, you are incentivized to learn that cooperation (filling a need) is the way to garner resources that give you the ability to get more done.
Second I’d point out that there are many voices currently arguing that the largest companies of today (FANG, often) have too much power and are monopolizing societies’ resources. I’d just like to contrast that with the same arguments made in earlier eras about IBM, Microsoft, AT&T, Standard Oil, big Hollywood, the railroad trusts, the A&P stores, and many others. If you have the right viewpoint, you can look at an economy where someone is earning more money than you think is just, and believe that they’re going to own the world in another year if nothing is done to stop them. It’s usually competition and innovation that keeps this from happening and not government or other concerted action.
I think this is probably correct to a degree—we do say we’re looking at AI takeover scenarios, and all but one of our scenarios (WFLL 1) eventually result in some terrible outcome for humanity and human values, like extinction. However, we do briefly discuss the possibility that the ‘takeover’ might be ambiguously bad—in WFLL 1 we said there was a 50:50 chance of actual human extinction, and in that scenario it’s even possible that something like modern human civilisation would continue. However, the fact that there’s some blurring at the edges does not mean that the possible scenarios fill up a continuous spectrum of badness. I think it’s quite likely that the possible outcomes are pretty bimodal, even if it’s not literally a binary of ‘we go extinct’ vs ‘techno-utopia’, we still either keep control of the future or lose more and more of it as time goes on. If we’re in a future where we’re cooperating and competing with TAIs in such a way that we can get what we want and influence the future, that’s not a takeover scenario.
I don’t disagree with this point and while there are differences of opinion my own view is that big tech isn’t a uniquely bad monopoly, i.e. not worse than things like Hollywood, Standard Oil etc. However, in this case, we have a unique reason to think that these problems will just keep getting worse—namely the presence of nonhuman actors that are power-seeking. So maybe the analogy that it will be like ‘big tech but worse’ doesn’t quite fit because there’s a specific reason as to why things would get much worse than any historical example of a monopoly.
This is one of the things we discuss in terms of alignment being ‘hackable’ - our term for how easy/hard it is to keep systems behaving well with changes to their incentives and incremental fixes. If alignment is quite hackable (i.e. there aren’t deep technical reasons systems will go wrong in ways that won’t be detected), then what you’re describing would work. If systems do the wrong thing we punish them by denying them the resources/rewards they’re seeking. AAFS/Production Web describe this strategy working for a while and then abruptly failing because of hidden vulnerabilities, because this post is about AI takeover stories, but it’s definitely possible that it doesn’t fail, and we end up in a multipolar outcome that’s persistently good.