In order to decide to not build it, all people who can and would otherwise build it must in some way end up not doing so. For any individual actor who could build it, they must either choose themselves to not build it, or be prevented from doing so. Pushing towards the former is why it’s a good idea to not publish ideas that could, even theoretically, help with building it. In order for the latter to occur, rules backed by sufficient monitoring and force must be used. I don’t expect that to happen in time. As a result, I am mostly optimistic about plans where it goes well, rather than plans where it doesn’t happen. Plans where it goes well depend on figuring out how to encode to it an indelible target that makes it care about everyone, and then convincing a team who will build it to use that target. as you can imagine, that is an extremely tall order. Therefore, I expect humanity to die, likely incrementally as more and more businesses grow that are more and more AI-powered and uninhibited by any worker or even owner constraints.
But those are the places where I see branches that can be intervened on. If you want to prevent it, people are attempting to get governments to implement rules sufficient to actually prevent it from coming into existence anywhere, at all. It looks to me like it’s going to just create regulatory capture and still allow the companies and governments to create catastrophically uncaring AI.
And no, your question is not the kind that would be deleted here. I appreciate you posting it. Sorry to be so harshly gloomy in response.
I, on the other hand, have very little confidence that people trying to build AGI will fail to quickly (within the next 3 years, aka 2027) find ways to do it. I do have confidence that we can politically coordinate to stop the situation becoming an extinction or near-extinction-level catastrophe. So I place much less emphasis on abstaining from publishing ideas which may help both alignment and capabilities, and more emphasis on figuring out ways to generate empirical evidence of the danger before it is too late, so as to facilitate political coordination.
I think that the situation in which humanity fails to politically coordinate to avoid building catastrophically dangerous AI is a situation that leads into conflict, likely a World War III with wide-spread use of nuclear weapons. I don’t expect humanity to go extinct from this and I don’t expect the rogue AGI to emerge as the victor, but I do think it is in everyone’s interests to work hard to avoid such a devastating conflict. I do think that any such conflict would likely wipe out the majority of humanity. That’s a pretty grim risk to be facing on the horizon.
In order to decide to not build it, all people who can and would otherwise build it must in some way end up not doing so. For any individual actor who could build it, they must either choose themselves to not build it, or be prevented from doing so. Pushing towards the former is why it’s a good idea to not publish ideas that could, even theoretically, help with building it. In order for the latter to occur, rules backed by sufficient monitoring and force must be used. I don’t expect that to happen in time. As a result, I am mostly optimistic about plans where it goes well, rather than plans where it doesn’t happen. Plans where it goes well depend on figuring out how to encode to it an indelible target that makes it care about everyone, and then convincing a team who will build it to use that target. as you can imagine, that is an extremely tall order. Therefore, I expect humanity to die, likely incrementally as more and more businesses grow that are more and more AI-powered and uninhibited by any worker or even owner constraints.
But those are the places where I see branches that can be intervened on. If you want to prevent it, people are attempting to get governments to implement rules sufficient to actually prevent it from coming into existence anywhere, at all. It looks to me like it’s going to just create regulatory capture and still allow the companies and governments to create catastrophically uncaring AI.
And no, your question is not the kind that would be deleted here. I appreciate you posting it. Sorry to be so harshly gloomy in response.
I, on the other hand, have very little confidence that people trying to build AGI will fail to quickly (within the next 3 years, aka 2027) find ways to do it. I do have confidence that we can politically coordinate to stop the situation becoming an extinction or near-extinction-level catastrophe. So I place much less emphasis on abstaining from publishing ideas which may help both alignment and capabilities, and more emphasis on figuring out ways to generate empirical evidence of the danger before it is too late, so as to facilitate political coordination.
I think that the situation in which humanity fails to politically coordinate to avoid building catastrophically dangerous AI is a situation that leads into conflict, likely a World War III with wide-spread use of nuclear weapons. I don’t expect humanity to go extinct from this and I don’t expect the rogue AGI to emerge as the victor, but I do think it is in everyone’s interests to work hard to avoid such a devastating conflict. I do think that any such conflict would likely wipe out the majority of humanity. That’s a pretty grim risk to be facing on the horizon.